Send patches - preferably formatted by git format-patch - to patches at archlinux32 dot org.
summaryrefslogtreecommitdiff
path: root/lib/libalpm/be_sync.c
AgeCommit message (Collapse)Author
2020-07-07Move signature payload creation to download engineAnatol Pomozov
Until now callee of ALPM download functionality has been in charge of payload creation both for the main file (e.g. *.pkg) and for the accompanied *.sig file. One advantage of such solution is that all payloads are independent and can be fetched in parallel thus exploiting the maximum level of download parallelism. To build *.sig file url we've been using a simple string concatenation: $requested_url + ".sig". Unfortunately there are cases when it does not work. For example an archlinux.org "Download From Mirror" link looks like this https://www.archlinux.org/packages/core/x86_64/bash/download/ and it gets redirected to some mirror. But if we append ".sig" to the end of the link url and try to download it then archlinux.org returns 404 error. To overcome this issue we need to follow redirects for the main payload first, find the final url and only then append '.sig' suffix. This implies 2 things: - the signature payload initialization need to be moved to dload.c as it is the place where we have access to the resolved url - *.sig is downloaded serially with the main payload and this reduces level of parallelism Move *.sig payload creation to dload.c. Once the main payload is fetched successfully we check if the callee asked to download the accompanied signature. If yes - create a new payload and add it to mcurl. *.sig payload does not use server list of the main payload and thus does not support mirror failover. *.sig file comes from the same server as the main payload. Refactor event loop in curl_multi_download_internal() a bit. Instead of relying on curl_multi_check_finished_download() to return number of new payloads we simply rerun the loop iteration one more time to check if there are any active downloads left. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-06-26Cleanup the old sequential download codeAnatol Pomozov
All users of _alpm_download() have been refactored to the new API. It is time to remove the old _alpm_download() functionality now. This change also removes obsolete SIGPIPE signal handler functionality (this is a leftover from libfetch days). Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
2020-05-09Swap alpm_db_update() implementation to multiplexed versionAnatol Pomozov
Now when all callers of the old alpm_db_update() function are gone we can remove this implementation. And then rename alpm_dbs_update() function to alpm_db_update(). Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09Implement multibar UIAnatol Pomozov
Multiplexed download requires ability to draw UI for multiple active progress bars. To implement it we use ANSI codes to move cursor up/down and then redraw the required progress bar. `pacman_multibar_ui.active_downloads` field represents the list of active downloads that correspond to progress bars. `struct pacman_progress_bar` is a data structure for a progress bar. In some cases (e.g. database downloads) we want to keep progress bars in order. In some other cases (package downloads) we want to move completed items to the top of the screen. Function `multibar_move_completed_up` allows to configure such behavior. Per discussion in the maillist we do not want to show download progress for signature files. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09Introduce event types for start/end database list downloadAnatol Pomozov
Multiplexed database/files downloads will use multiple progress bars. The UI logic is quite complicated and printing error messages while handling multiple progress bars is going to be challenging. Instead we are going to save all ALPM error messages to a list and flush it at the end of the download process. Use on_progress variable that blocks error messages printing. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09Introduce alpm_dbs_update() function for parallel db updatesAnatol Pomozov
This is an equivalent of alpm_db_update but for multiplexed (parallel) download. The difference is that this function accepts list of databases to update. And then ALPM internals download it in parallel if possible. Add a stub for _alpm_multi_download the function that will do parallel payloads downloads in the future. Introduce dload_payload->filepath field that contains url path to the file we download. It is like fileurl field but does not contain protocol/server part. The rationale for having this field is that with the curl multidownload the server retry logic is going to move to a curl callback. And the callback needs to be able to reconstruct the 'next' fileurl. One will be able to do it by getting the next server url from 'servers' list and then concat with filepath. Once the 'parallel download' refactoring is over 'fileurl' field will go away. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-04-13Use GOTO_ERR throughoutAllan McRae
The GOTO_ERR define was added in commit 80ae8014 for use in future commits. There are plenty of places in the code base it can be used, so convert them. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-04-13Add REALLOC macro to simplify realloc error handlingRikard Falkeborn
realloc can fail just like the other memory allocation functions. Add a macro to simplify handling of realloc failures, similar to the already existing MALLOC, CALLOC, etc. Replace the existing realloc uses with the new macro, allowing us to move tedious error handling to the macro. Also, in be_package and be_sync, this fixes hypothetical memory leaks (and thereafter null pointer dereferences) in case realloc fails to shrink the allocated memory. Signed-off-by: Rikard Falkeborn <rikard.falkeborn@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-02-10build-aux/update-copyright 2019 2020Allan McRae
Signed-off-by: Allan McRae <allan@archlinux.org>
2020-01-28Docs docs docsmorganamilo
libalpm: move docs from .c files into alpm.h And fix/expand some along the way. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-01-27Increase maximum database sizeAllan McRae
We previously has the maximum database size as 25MB. This was set in the days before repos had as many packages as they do now, and before we started distributing files databases. Increase this limit to 128MB. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-01-07Use c99 struct initialization to avoid memset callsDave Reisner
This is guaranteed less error prone than calling memset and hoping the human gets the argument order correct.
2019-10-23Update copyright yearsAllan McRae
make update-copyright OLD=2018 NEW=2019 Signed-off-by: Allan McRae <allan@archlinux.org>
2019-03-07Remove support for deltas from libalpmAllan McRae
Signed-off-by: Allan McRae <allan@archlinux.org>
2019-01-10libalpm: parse {check, make}depends when reading databasemorganamilo
Commit 0994893b0e6b627d45a63884ac01af7d0967eff2 added the alpm_pkg_get_{make,check}depends functions but forgot to include logic for parsing these fields from the database. As a result these functions will always return an empty list. This commit adds the parsing logic. Signed-off-by: morganamilo <morganamilo@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2018-11-27always allow explicit empty siglevel for sync dbsAndrew Gregory
An empty siglevel does not do any signature verification which is exactly what we want when compiled without gpg support. This is already allowed in other parts of the codebase and required for the test suite to pass when compiled without gpg support. Fixes: FS#60880 Signed-off-by: Andrew Gregory <andrew.gregory.8@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2018-11-27add specific error for missing gpg supportAndrew Gregory
"wrong or NULL argument passed" is a useless error for end users. Fixes FS#60880. Signed-off-by: Andrew Gregory <andrew.gregory.8@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2018-05-14Remove all modelines from the projectEli Schwartz
Many of these are pointless (e.g. there is no need to explicitly turn on spellchecking and language dictionaries for the manpages by default). The only useful modelines are the ones enforcing the project coding standards for indentation style (and "maybe" filetype/syntax, but everything except the asciidoc manpages and makepkg.conf is already autodetected), and indent style can be applied more easily with .editorconfig Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> Signed-off-by: Allan McRae <allan@archlinux.org>
2018-03-14Update coyrights for 2018Allan McRae
make update-copyright OLD=2017 NEW=201 Signed-off-by: Allan McRae <allan@archlinux.org>
2018-01-10add missing newline to debug messageAndrew Gregory
Signed-off-by: Andrew Gregory <andrew.gregory.8@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2018-01-10detect pkghash allocation failureAndrew Gregory
If rehash ever failed with a full hash it would return the old hash that is already full. get_hash_position would then loop forever because it would never find an empty bucket. Signed-off-by: Andrew Gregory <andrew.gregory.8@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2018-01-10Do not continuously try to open an invalid databaseAllan McRae
If you manage to download a bad database (e.g. an html file when behind a proxy or with a badly configured webserver), pacman makes sure you know about it. Here is some example output: error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format I don't know how many times that gets printed because it goes beyond my scrollback buffer. Flag a database that we can "open" and "fstat" but not read from as invalid to avoid this. Signed-off-by: Allan McRae <allan@archlinux.org>
2018-01-06avoid printing NULL stringAndrew Gregory
Signed-off-by: Andrew Gregory <andrew.gregory.8@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2017-05-08be_sync: error out if a db cannot be parsedAndrew Gregory
Signed-off-by: Andrew Gregory <andrew.gregory.8@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2017-01-04add _alpm_filelist_sortAndrew Gregory
Signed-off-by: Andrew Gregory <andrew.gregory.8@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2017-01-04Update copyright yearsAllan McRae
Signed-off-by: Allan McRae <allan@archlinux.org>
2016-10-22Represent bitfields as ints, not enumsIvy Foster
Many bitfield variables are declared to be enums, because they are generated using bitwise operations on enums such. However, their actual values aren't necessary members of their parent enum, so declaring them 'int' is more accurate. Signed-off-by: Ivy Foster <ivy.foster@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2016-10-22Add ALPM_ERR_OK to _alpm_errno_tIvy Foster
This allows functions which return an _alpm_errno_t to always return a genuine _alpm_errno_t for consistency, even in cases where there are no errors. Since ALPM_ERR_OK = 0, their callers can still simply check 'err = some_fn(); if (!err) { ... }'. Signed-off-by: Ivy Foster <ivy.foster@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2016-09-25Change type of count in be_syncRikard Falkeborn
Making it size_t matches the return value of alpm_list_count() and avoids the implicit cast to int. Signed-off-by: Rikard Falkeborn <rikard.falkeborn@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2016-09-25Return boolean from db_populateRikard Falkeborn
Since the number of packages is not used anywhere, just return a boolean to avoid the implicit cast from size_t to int in be_local.c. Use 0 as success to be consistent with db_validate. Signed-off-by: Rikard Falkeborn <rikard.falkeborn@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2016-09-09lib/libalpm/be_sync.c: Close memory leaks when mallocing while out of memoryIvy Foster
Signed-off-by: Ivy Foster <ivy.foster@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2016-01-04Update copyright years for 2016Allan McRae
make update-copyright OLD=2015 NEW=2016 Signed-off-by: Allan McRae <allan@archlinux.org>
2015-12-06db_update: always clear db flags after updateAndrew Gregory
Signature downloading and DB validation was being based on the most recent download status for the DB. If a DB successfully downloaded but a signature did not, db_update would move to the next server. If the next server tried does not have a more recent copy of the DB, db_update would not download the DB again and would forget that the DB had previously been updated. In this case it would skip validation entirely, leaving an updated DB with the original validation status. Signed-off-by: Andrew Gregory <andrew.gregory.8@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2015-11-03ensure realloc has a positive sizeAndrew Gregory
If given size 0 POSIX allows realloc to return a pointer that is not suitable for use. Signed-off-by: Andrew Gregory <andrew.gregory.8@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2015-08-08libalpm: Force update when database is marked as corrupted.Sebastian Lackner
When a database and its signature is updated non-atomically on a server, there is a window where a user may update the database but grab the old signature. The database is marked as invalid by libalpm, which can be fixed by forcing a refresh (assuming the server has caught up and the user realizes what has happened) or with a future update of the repo. Work around this by forcing a repository refresh whenever a database is invalid. Signed-off-by: Sebastian Lackner <sebastian@fds-team.de> Signed-off-by: Allan McRae <allan@archlinux.org>
2015-07-15Read file lists from databasesAllan McRae
If a sync database contains a "files" file, the file list will be read. Currently, there is no known demand for the file list to be lazy loaded by any libalpm frontend, so these files are read whenever present. Lazy loading can be implemented when a demand exists. Signed-off-by: Allan McRae <allan@archlinux.org>
2015-07-15Allow frontends to specify the sync database extensionAllan McRae
This allows frontends to select between the .db and .files databases currently supplied by repo-add or any other compatible database. Signed-off-by: Allan McRae <allan@archlinux.org>
2015-05-12libalpm: Do a sanity check before manipulating final DB URLDavid Macek
The change in commit 9d96bed9d6b57 causes download errors for the .db.sig file in case the final URL for the .db file contains query strings or other unexpected stuff. This commit isn't intended to be a total solution, but it should eliminate the problem in the most obvious cases. Signed-off-by: Allan McRae <allan@archlinux.org>
2015-05-12Fix comment in alpm_db_updateDavid Macek
Fix comment to better explain the magic constant used when allocating a buffer for ".db.sig" URL. Signed-off-by: Allan McRae <allan@archlinux.org>
2015-02-01Update copyright notices for 2015Allan McRae
Signed-off-by: Allan McRae <allan@archlinux.org>
2015-01-21conf.c: use masks for siglevel inheritanceAndrew Gregory
This will allow pacman to parse its config file in a single pass and removes the need for the *_SET siglevels in alpm that were only required for pacman's siglevel inheritance. Signed-off-by: Andrew Gregory <andrew.gregory.8@gmail.com>
2014-12-24libalpm: Parse and write PKGBASEsJohannes Löthberg
This commit adds support to libalpm to parse the pkgbase present in packages .PKGINFO files, writing the PKGBASE to the %BASE% section of the local DBs desc files and for parsing it again when loading the local DB Signed-off-by: Johannes Löthberg <johannes@kyriasis.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2014-11-17Fix crash when using external downloaderAllan McRae
Commit 9d96bed9 attempts to use the same effective URL for the db and its signature download. However, this information is not available when we use an external downloader, resulting in a crash. Fall back to the old method when the effective URL is unavailable. Signed-off-by: Allan McRae <allan@archlinux.org>
2014-10-19dload: mark final_url as constChristian Hesse
Signed-off-by: Allan McRae <allan@archlinux.org>
2014-10-16be_sync: use effective URL for db file signature downloadChristian Hesse
If download server is dynamic mirror chances are that db file download and db file signature download are redirected to different mirrors, resulting in invalid signature. This uses effective URL for db file signature download and makes the files always match. Signed-off-by: Christian Hesse <mail@eworm.de> Signed-off-by: Allan McRae <allan@archlinux.org>
2014-09-30libalpm: export alpm_splitdep as alpm_dep_from_depstring and alpm_dep_freeFlorian Pritz
Signed-off-by: Florian Pritz <bluewind@xinu.at>
2014-08-04handle_unlock: log lock removal failureAndrew Gregory
Rather than have individual callers log failure, just do it directly in _alpm_handle_unlock. Signed-off-by: Andrew Gregory <andrew.gregory.8@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2014-03-03Update the event callbackOlivier Brunel
Instead of using two void* arguments for all events, we now send one pointer to an alpm_event_t struct. This contains the type of event that was triggered. With this information, the pointer can then be typecasted to the event-specific struct in order to get additional arguments. Signed-off-by: Olivier Brunel <jjk@jjacky.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2014-01-28Remove ts and sw from vim modeline when noet is setFlorian Pritz
Forcing vim users to view files with a tabstop of 2 seems really unnecessary when noet is set. I find it much easier to read code with ts=4 and I dislike having to override the modeline by hand. Command run: find . -type f -exec sed -i '/vim.* noet/s# ts=2 sw=2##' {} + Signed-off-by: Florian Pritz <bluewind@xinu.at> Signed-off-by: Allan McRae <allan@archlinux.org>
2014-01-06Update copyright years for 2014Allan McRae
Signed-off-by: Allan McRae <allan@archlinux.org>

Copyright © 2002-2022 Judd Vinet and Aaron Griffin. Copyright © 2022 Erich Eckner.
The Arch Linux name and logo are recognized trademarks. Some rights reserved. The registered trademark Linux® is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis.