Skip to content

Split databases into branches/flagged (and various)

Carlo Landmeter requested to merge github/fork/clandmeter/split_db into master
  • split databases into branches
  • move flagged table to separate db read-write
  • set branch databases as read-only (from webif)
  • change relation of pkg vs flagged status
  • added migration script to migrate to new flagged table format
  • added new provider_priority field to apk header
  • add settings for default branch and arch
  • make edge the default branch in packages list
  • add SQL debug option in config
  • remove uneeded usage of class
  • move all utils to utils.lua
  • optimize SQL queries
  • re-factor importer script
  • allow to change http listening port via environment

This commit changes how we handle databases. Instead of having everything in a single sqlite database we split each branch into its own sqlite database. This will limit the amount of rows in the pkg/file per table and improve performance.

Flagged table has also been moved to its own database and has been redesigned. Splitting it into its own db makes it possible to open the branch databases read-only. This redesign creates a new composite index (repo,origin,version) so the packages table is now related to this index which makes it possible to remove the current flag id column. This also introduced the updated column to be able to unflag a package and make it unlisted on flagged listing but still be able to show the status on individual packages when some archs are behind. Every time packages are updated the unique triple will be looked up in the flagged table and updated/unflagged when found.

Migration is handled by the migrate_v2.lua script. It needs to be run against the old single aports.db and will generate a single flagged.db after which the import script can be run to start generating the splitted aports-branch.dbs. Some settings need to be updated from the provided sample config ie the db location is changed to a path instead of a file and the apkindex format is now defined in config instead of hard coded in multiple places.

Merge request reports