| Name: sqlite |
| URL: https://sqlite.org/ |
| Version: 3.30.1-patched |
| Included In Release: Yes |
| Security Critical: Yes |
| License: Public domain |
| |
| 1) Managing differences between SQLite core and Chromium's version. |
| 2) Making changes to Chromium SQLite. |
| 3) Import new release of SQLite. |
| 4) Running SQLite's test suite within the Chromium checkout. |
| |
| --- |
| |
| 1) Managing differences between SQLite core and Chromium's version. |
| |
| Chromium maintains some differences WRT SQLite, for reasons beyond this |
| document's remit. Some differences are bugs we have found and fixed (and |
| hopefully upstreamed), some are fixes we've backported from a later version of |
| SQLite, and some our local changes unlikely to ever be upstreamed. New versions |
| of SQLite are imported every year or two, at which point the changes need to be |
| reviewed for continued applicability, and sometimes adjusted to reflect upstream |
| code changes. |
| |
| To this end, the repository contains a reference copy of the SQLite source code |
| as of the last import, plus a series of patches which can be applied to |
| re-create the current trunk code. These patches are generated and processed by |
| git, with the intention of re-creating a commit series so that importers can use |
| their regular revision-control knowledge to manage import merges. |
| |
| The directory structure is as follows. Files common to all third_party projects |
| (BUILD.GN, OWNERS, LICENSE) are omitted. |
| |
| * sqlite-src-*/ - Upstream source code, without any modifications. The number |
| after src- is a release version or a snapshot number (which is |
| a commit time). See https://www.sqlite.org/cgi/src/timeline |
| * patches/ - Our patches to the currently used release, formatted by git using |
| the UNIX mailbox format. The patches can be applied with git am, |
| and created with git format-patch. |
| * patched/ - The currently used source code, with our patches applied. |
| * amalgamation/ - The supported method of using SQLite is via an amalgamation |
| build, which merges all the code in one .c file and one .h |
| file. See https://www.sqlite.org/amalgamation.html |
| * amalgamation/config.h - Linux build configuration |
| * scripts/ - Scripts that generate the files in the amalgamation |
| * sqlite.h - The header used by the rest of Chromium to include SQLite. This |
| forwards to amalgamation/sqlite3.h |
| * fuzz/ - Google OSS-Fuzz (ClusterFuzz) testing for Chromium's SQLite build |
| |
| --- |
| |
| 2) Making changes to Chromium SQLite. |
| |
| third_party/sqlite/patched is the patched source from SQLite. This is used to |
| generate the amalgamation, a concatenation of all of the files into a giant |
| sqlite3.c. To prototype, edit in patched/, then call |
| ./scripts/generate_amalgamation.sh |
| to regenerate sqlite3.c. The code in patched/ is much easier to edit, and the |
| SQLite test framework can easily be run. During development it may be |
| convenient to modify BUILD.gn based on patched/main.mk to just pull in the |
| patched/ files rather than sqlite3.c. |
| |
| Once your patch is complete, squash it down into a reasonable CL, then |
| re-generate the patches. This is a truncated version of the import flow. The |
| following is written like a shell script to allow copy/paste to a shell, ignore |
| comments and change the obvious lines. These instructions should work on Linux |
| or OSX. They may assume a modern version of git. |
| |
| # The steps below are easier if done in the SQLite directory. |
| cd third_party/sqlite |
| |
| # Must match the version in //third_party/sqlite/sqlite-src-xxxxxxx. |
| # This is SQLite's version number, and uses upstream's convention. |
| export BASE=3300100 |
| export GNU_SED=sed # OSX: "brew install gnu-sed", then use "gsed" here. |
| |
| #### Create a reference branch. |
| git new-branch sqlite-base |
| git rm -rf patched |
| cp -r sqlite-src-${BASE}/ patched |
| # Clean up trailing whitespace and CRLF so any patches look clean. |
| find patched/ -type f -not -iname "*.db" -not -iname "*.eps" \ |
| -not -iname "*.ico" -not -iname "*.jpg" -not -iname "*.pfx" \ |
| -not -iname "*.png" -not -iname "*.tiff" -not -iname "*.vsix" \ |
| -exec ${GNU_SED} --in-place 's/[[:space:]]\+$//' {} \+ |
| git add patched/ |
| git clean -i -d -x patched # Make sure no file is git-ignored. |
| git commit -m "Squash: Reset SQLite patched/ to sqlite-src-${BASE}." |
| # This branch will not build. It will be used for rebasing, then deleted. |
| |
| #### Create a reference branch with patches applied. |
| git new-branch --upstream-current sqlite-dev |
| git am --keep-non-patch --ignore-space-change patches/*.patch |
| git diff origin/master patched/ |
| # This branch should be identical to master. |
| |
| #### Develop and validate the change, or cherry-pick it from a dev branch. |
| # The goal is to have a set of reasonably-independent CLs which can be |
| # understood separately, so that future importers can sensibly determine how to |
| # handle conflicts. So use git-rebase and slipstream fixups back into existing |
| # patches, or add a new patch. |
| ./scripts/generate_amalgamation.sh |
| git cl format amalgamation/rename_exports.h |
| cd ../.. |
| ninja -C out/Default |
| # Check that extract_sqlite_api.py added chrome_ to all exported symbols. |
| # Only "_fini" and "_init" should be unprefixed. |
| nm -B out/Default/libchromium_sqlite3.so | cut -c 18- | sort | grep '^T' |
| out/Default/sql_unittests |
| third_party/blink/tools/run_web_tests.py -t Default storage/websql/ |
| cd third_party/sqlite |
| |
| #### Create the review. |
| # Rebuild the patch set. |
| git rm patches/* |
| git format-patch --output-directory=patches --zero-commit \ |
| sqlite-base..sqlite-dev |
| |
| ### Document and link any backported patches to the upstream repository URL |
| ### and crbug. |
| # Under the *.patch's added by this change, the below information should be |
| # added between the "Subject: *" line and the first "---" line. For an example, |
| # please reference https://crrev.com/c/1480822/2/third_party/sqlite/patches/0008-Adjustments-to-the-page-cache-to-try-to-avoid-harmle.patch. |
| # |
| # This backports https://www.sqlite.org/src/info/XXX |
| # |
| # Bug: XXX |
| |
| ### Commit and create CL |
| git add amalgamation/ |
| git add patches/ |
| git commit -m "Squash: regenerate amalgamation and patches." |
| git branch --set-upstream-to=origin/master |
| git cl upload --squash |
| |
| --- |
| |
| 3) Import a new SQLite release. |
| |
| Importing a new SQLite involves merging our local changes with SQLite's changes. |
| Like any other merge, this may involve modifying some commits and dropping |
| others. The basic idea below is to generate git branches to work with: |
| |
| * sqlite-new-upstream - new release code archived in a separate directory |
| * sqlite-old-base - current release without patches |
| * sqlite-old - current release with patches mapped to git commits |
| * sqlite-new-base - new release without patches |
| * sqlite-new - new release with patches mapped to git commits |
| * sqlite-new-cl - new release in one git commit, for git cl upload |
| |
| We will upload sqlite-new-upstream as a massive (800k LOC+) CL that cannot |
| possibly be reviewed, but is generated in an automated fashion. We will then |
| squash sqlite-new to sqlite-new-upstream and obtain one CL that only contains |
| diffs. The second CL is still large, but it's a fraction of the first |
| (automated) CL. |
| |
| # The steps below are easier if done in the SQLite directory. |
| cd third_party/sqlite |
| |
| # The numbers below are SQLite version numbers, and use upstream's convention |
| # for tagging release binaries and source zipballs. |
| export OLD=3300100 |
| export NEW=3310000 |
| export GNU_SED=sed # OSX: "brew install gnu-sed", then use "gsed" here. |
| |
| #### Download and unpack the new SQLite release. |
| git new-branch sqlite-new-upstream |
| # URL from "Alternative Source Code Formats" at https://sqlite.org/download.html |
| curl https://sqlite.org/2019/sqlite-src-${NEW}.zip > upstream.zip |
| mkdir sqlite-src-${NEW} |
| unzip ./upstream.zip -d sqlite-src-${NEW} |
| rm ./upstream.zip |
| mv sqlite-src-${NEW}/sqlite-*/* sqlite-src-${NEW}/ |
| rm -r sqlite-src-${NEW}/sqlite-*/.fossil-settings |
| rmdir sqlite-src-${NEW}/sqlite-*/ |
| rm -r sqlite-src-${NEW}/compat |
| xdg-open sqlite-src-${NEW} # Make sure everything looks right. |
| |
| #### Add the new release code in a separate CL, for code review sanity. |
| git add sqlite-src-${NEW} # Committing the code as downloaded, on purpose. |
| git clean -i -d -x sqlite-src-${NEW} # Make sure no file is git-ignored. |
| git commit -m "sqlite: Add code for release ${NEW}" |
| git cl upload # Have the new code in a separate CL. |
| |
| #### Create a branch for the old SQLite release's upstream version. |
| git new-branch sqlite-old-base |
| git rm -rf patched |
| cp -r sqlite-src-${OLD}/ patched |
| # Clean up trailing whitespace and CRLF so any patches look clean. |
| find patched/ -type f -not -iname "*.db" -not -iname "*.eps" \ |
| -not -iname "*.ico" -not -iname "*.jpg" -not -iname "*.pfx" \ |
| -not -iname "*.png" -not -iname "*.tiff" -not -iname "*.vsix" \ |
| -exec ${GNU_SED} --in-place 's/[[:space:]]\+$//' {} \+ |
| git add patched/ |
| git clean -i -d -x patched # Make sure no file is git-ignored. |
| git commit -m "Squash: Reset SQLite patched/ to sqlite-src-${OLD}." |
| # This branch will not build. It will be used for rebasing, then deleted. |
| |
| #### Create a branch for our old SQLite code, with patches mapped to commits. |
| git new-branch --upstream-current sqlite-old |
| git am --keep-non-patch --ignore-space-change patches/*.patch |
| git diff origin/master patched/ |
| # This branch should be identical to master. |
| |
| #### Create a branch for the new SQLite release's upstream version. |
| git checkout sqlite-old-base |
| git new-branch --upstream-current sqlite-new-base |
| git rm -rf patched |
| git checkout sqlite-new-upstream -- sqlite-src-${NEW}/ |
| git mv sqlite-src-${NEW}/ patched |
| # Clean up trailing whitespace and CRLF so any patches look clean. |
| find patched/ -type f -not -iname "*.db" -not -iname "*.eps" \ |
| -not -iname "*.ico" -not -iname "*.jpg" -not -iname "*.pfx" \ |
| -not -iname "*.png" -not -iname "*.tiff" -not -iname "*.vsix" \ |
| -exec ${GNU_SED} --in-place 's/[[:space:]]\+$//' {} \+ |
| git add patched/ |
| git clean -i -d -x patched # Make sure no file is git-ignored. |
| git commit -m "Squash: Reset SQLite patched/ to sqlite-src-${NEW}." |
| # This branch will not build. It will be used for rebasing, then deleted. |
| |
| #### Create a branch for updating our patches. |
| git checkout sqlite-old |
| git new-branch --upstream-current sqlite-new |
| # Rebase our patches, which are mapped to separate commits, onto the new |
| # release. There will be merge conflicts that must be fixed. This is the |
| # interesting part of the work. |
| git rebase sqlite-new-base |
| |
| #### Finally, create the branch that we'll upload. |
| git new-branch --upstream-current sqlite-new-cl |
| ./scripts/generate_amalgamation.sh |
| git cl format amalgamation/rename_exports.h |
| |
| #### Copy any new entries to the seed-corpus for dbfuzz2. |
| # We use a set of seed databases for fuzzing SQLite's resilience to database |
| # corruption. Sometimes, new seed databases are added upstream. |
| # Find any files of the pattern patched/test/dbfuzz2-seed*.db, and copy them to |
| # the fuzz/db_corpus directory. |
| cp --no-clobber patched/test/dbfuzz2-seed*.db fuzz/db_corpus |
| |
| #### Validate the upgrade. |
| # The goal is to have a set of reasonably-independent CLs which can be |
| # understood separately, so that future importers can sensibly determine how to |
| # handle conflicts. So use git-rebase and slipstream fixups back into their |
| # original CL until everything builds and works. |
| cd ../.. |
| autoninja -C out/Default |
| # Check that extract_sqlite_api.py added chrome_ to all exported symbols. |
| # Only "_fini" and "_init" should be unprefixed. |
| nm -B out/Default/libchromium_sqlite3.so | cut -c 18- | sort | grep '^T' |
| out/Default/sql_unittests |
| third_party/blink/tools/run_web_tests.py -t Default storage/websql/ |
| cd third_party/sqlite |
| |
| #### Create the review. |
| # Rebuild the patch set. |
| git rm patches/* |
| git format-patch --output-directory=patches --zero-commit \ |
| sqlite-new-base..sqlite-new |
| git add amalgamation/ |
| git add patches/ |
| git commit -m "Squash: regenerate amalgamation and patches." |
| git branch --set-upstream-to=origin/master |
| git cl upload --squash |
| # Example commit message (https://crrev.com/c/1699400): |
| # sqlite: Upgrade from 3.28.0 to 3.29.0 |
| # |
| # sqlite 3.29.0 update changes: |
| # https://www.sqlite.org/releaselog/3_29_0.html |
| # |
| # Bug: 983713 |
| |
| #### Drop the old version of SQLite. |
| git new-branch sqlite-rm-old |
| git rm -r sqlite-src-${OLD} |
| git commit -m "sqlite: Remove source code for old release ${OLD}." |
| git cl upload |
| |
| Note that things can be broken down differently, if you prefer. For instance, |
| adding the new version of the SQLite distro and removing the old one can be |
| distinct CLs. |
| |
| -------------------------------------------- |
| |
| 4) Running SQLite's test suite within the Chromium checkout. |
| |
| TODO(pwnall): This hasn't been tried out for at least a year. |
| |
| Prerequisites: The test suite requires tcl-dev and libicu-dev. Install those on |
| Ubuntu like: |
| sudo apt-get install tcl8.6-dev libicu-dev |
| On macOS, I use Homebrew: |
| brew install icu4c tcl-tk |
| export PATH="$(brew --prefix icu4c)/bin:$(brew --prefix tcl-tk)/bin:$PATH" |
| |
| Run the commands in scripts/generate_amalgamation.sh, but replace the "make" |
| command with "make test". |
| |
| make test > /tmp/test.log |
| egrep 'errors out of' /tmp/test.log |
| # Show broken tests: |
| egrep 'Failures on these tests:' /tmp/test.log |
| # Broken tests will also show lines ending in "..." instead of "... Ok". |
| |
| In version 3.10.2 on OSX 10.11.2, I see: |
| 6 errors out of 139819 tests |
| The failed tests are: |
| pager4-1.3 pager4-1.4 pager4-1.5 pager4-1.9 pager4-1.10 pager4-1.11 |
| This is due to the change in os_unix.c fileHasMoved() to support WebDatabase. |
| Commenting out the early return allows them to succeed. |
| |
| In version 3.10.2 on Ubuntu 14.04.3 I see: |
| 9 errors out of 140309 tests |
| The failed tests are: |
| oserror-1.1.1 oserror-1.1.2 oserror-1.1.3 |
| pager4-1.3 pager4-1.4 pager4-1.5 pager4-1.9 pager4-1.10 pager4-1.11 |
| The oserror tests fail because there are too many fds available, and can be |
| fixed by running "ulimit -n 1024" before the test. The pager4 tests are failing |
| for the same reason as for OSX. |
| |
| -- |
| |
| NOTE(pwnall): On Ubuntu it is possible to run the tests in a tmpfs something |
| like: |
| |
| TMPFS=/dev/shm/sqlite_build |
| BUILD=$PWD |
| mkdir $TMPFS |
| (cd $TMPFS ; $BUILD/testfixture $BUILD/../test/veryquick.test >/tmp/test.log) |
| |
| This is faster, but it is plausible that different things are being tested than |
| real-world use. |