Send patches - preferably formatted by git format-patch - to patches at archlinux32 dot org.
summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)Author
2020-12-21doc: remove stray doxygen commentmorganamilo
Signed-off-by: Allan McRae <allan@archlinux.org>
2020-12-09Implement TotalDownload functionalityAnatol Pomozov
With the recent 'multibar' interface changes TotalDownload has been disabled. Now we have a new UI and we need to find another way to display this information. When 'TotalDownload' config option is enabled we are going to have an extra progress bar at the bottom of the screen that shows how much of the entire download has been completed. Closes FS#68202 Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-12-03libalpm: set ret in download filesmorganamilo
download_files never set ret on failiure, so even when downloading fails, the transaction goes on to commit and error out. :: Retrieving packages... python-packaging-20.4-4-any.pkg.tar.zst failed to download error: failed retrieving file 'python-packaging-20.4-4-any.pkg.tar.zst' from mirror.oldsql.cc : The requested URL returned error: 404 warning: failed to retrieve some files (1/1) checking keys in keyring (1/1) checking package integrity error: failed to commit transaction (wrong or NULL argument passed) Errors occurred, no packages were upgraded. Also make the ret checking more consistent. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-11-26libalpm: add alpm_option_get_parallel_downloadsmorganamilo
Signed-off-by: Allan McRae <allan@archlinux.org>
2020-11-26libalpm: set errno when setting parallel downloads with an invalid numbermorganamilo
Signed-off-by: Allan McRae <allan@archlinux.org>
2020-11-26libalpm: clarify alpm_download_event_completed_t statusmorganamilo
The comment makes it seem that the result itself is an error code. But all it does is simply return -1 to indicate an error occured; Signed-off-by: Allan McRae <allan@archlinux.org>
2020-11-26libalpm: don't call dlcb when not setmorganamilo
Fixes FS#68728: Signed-off-by: Allan McRae <allan@archlinux.org>
2020-11-26libalpm: set parallel_downloads to 1 when creating the handlemorganamilo
Fixes FS#68729 Signed-off-by: Allan McRae <allan@archlinux.org>
2020-09-23FS#66472: Remove *.sig file if package corruptedAnatol Pomozov
In case if a package corrupted (e.g. signature or hash is invalid) pacman tries to remove the package file to redownload it anew the next time. Remove *.sig file as well to make sure no data is left for the invalid package. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-09-03remove more autotools filesEli Schwartz
We forgot to remove m4/ in commit 454ea024383eab60295e4c4fdf2c329475887b2c and now it's tragically reminding me of autotools! Also take this opportunity to drop some symlinks in lib/libalpm/ for libcommon source files. In autotools these were built specifically for libalpm and needed to be available in that directory, but the meson setup just has libalpm depend on libcommon. So these pseudo source files aren't needed anymore. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-08-10Enable sha256/md5 hash verification if detached signatures are usedAnatol Pomozov
Pacman has multiple ways to verify package content integrity: - gpg signature - sha256 - md5 These verification mechanisms overlap each other. gpg signatures already contain hash value of the package content. So if a package signature is present then pacman ignored the other 2 hash values. This worked well with signtures embedded into pacman database. Recently pacman got an ability to handle detached signatures (*.sig files located next to the package files). If pacman verifies detached signature only then one can replace pkg+sig files with some other content and pacman still processes it as a valid package. To prevent it we need to verify database<->package integrity using hash values stored in the database. This commit fixes FS#67232 The new debug output is: checking package integrity... debug: found cached pkg: /var/cache/pacman/pkg/ruby-2.7.1-2-x86_64.pkg.tar.zst debug: sha256sum: 77baf61c62c5570b3a37cf0c3b16c5d9a97dde6fedd1a3528bf0cc5f96dd5e52 debug: checking sha256sum for /var/cache/pacman/pkg/ruby-2.7.1-2-x86_64.pkg.tar.zst debug: sig data: <from .sig> debug: checking signature for /var/cache/pacman/pkg/ruby-2.7.1-2-x86_64.pkg.tar.zst debug: 1 signatures returned debug: fingerprint: B5971F2C5C10A9A08C60030F786C63F330D7CB92 Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-08-10Fix error during keyring checkingAnatol Pomozov
With current master version the 'keyring checking' step produces an error: debug: returning error 6 from alpm_pkg_get_sig (../lib/libalpm/package.c: 274) : wrong or NULL argument passed The package signature is still checked later at the integrity verification step though. This commit fixes keyring checking and now the debug log looks like this: debug: found cached pkg: /var/cache/pacman/pkg/ruby-2.7.1-2-x86_64.pkg.tar.zst debug: found detached signature /var/cache/pacman/pkg/ruby-2.7.1-2-x86_64.pkg.tar.zst.sig with size 566 debug: found signature key: 786C63F330D7CB92 debug: looking up key 786C63F330D7CB92 locally debug: key lookup success, key exists Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-07-14Check that destfile_name exists before using itAnatol Pomozov
In some cases (when trust_remote_name is used for a URL without a filename and no Content-Disposition is provided by the server) destfile_name will be NULL. In this case payload data will be stored in tempfile_name and no destfile_name is set. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-07-14Do not free payload fields in the middle of this structure useAnatol Pomozov
At the end of payload use it calls _alpm_dload_payload_reset() that will free() these and other fields anyway. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-07-14Build signature remote name based on the main payload nameAnatol Pomozov
The main payload final name might be affected by url redirects or Content-Disposition HTTP header value. We want to make sure that accompanion *.sig filename always matches the package filename. So ignore finalname/Content-Disposition for the *.sig file. It also helps to fix a corner case when the download URL does not contain a filename and server provides Content-Disposition for the main payload but not for the signature payload. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-07-07Fallback to detached signatures during keyring checkAnatol Pomozov
Pacman has a 'key in keyring' verification step that makes sure the signatures have a valid keyid. Currently pacman parses embedded package signatures only. Add a fallback to detached signatures. If embedded signature is missing then it tries to read corresponding *.sig file and get keyid from there. Verification: debug: found cached pkg: /var/cache/pacman/pkg/glib-networking-2.64.3-1-x86_64.pkg.tar.zst debug: found detached signature /var/cache/pacman/pkg/glib-networking-2.64.3-1-x86_64.pkg.tar.zst.sig with size 310 debug: found signature key: A5E9288C4FA415FA debug: looking up key A5E9288C4FA415FA locally debug: key lookup success, key exists Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-07-07FS#33992: force download *.sig file if it does not exist in the cacheAnatol Pomozov
In case if *.pkg exists but *.sig file does not we still have to pass the pkg to multi_download API. To avoid redownloading *.pkg file we use CURLOPT_TIMECONDITION curl option. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-07-07Add a utility function to check whether a file exists in the cacheAnatol Pomozov
It is similar to _alpm_filecache_find() but does not return a dynamically allocated memory to user. Thus the user does not need to free this resource. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-07-07Do not use counter for error trackingAnatol Pomozov
Current code uses an incrementing counter to check whether a function returned error: errors += some_function(); if(errors) { goto finish } Replace with a more standard variable errors = some_function(); if(errors) { goto finish } Rename 'errors' variable to a more typical 'ret'. Avoid reporting both ALPM_EVENT_PKG_RETRIEVE_FAILED and ALPM_EVENT_PKG_RETRIEVE_DONE in the error path. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-07-07Move signature payload creation to download engineAnatol Pomozov
Until now callee of ALPM download functionality has been in charge of payload creation both for the main file (e.g. *.pkg) and for the accompanied *.sig file. One advantage of such solution is that all payloads are independent and can be fetched in parallel thus exploiting the maximum level of download parallelism. To build *.sig file url we've been using a simple string concatenation: $requested_url + ".sig". Unfortunately there are cases when it does not work. For example an archlinux.org "Download From Mirror" link looks like this https://www.archlinux.org/packages/core/x86_64/bash/download/ and it gets redirected to some mirror. But if we append ".sig" to the end of the link url and try to download it then archlinux.org returns 404 error. To overcome this issue we need to follow redirects for the main payload first, find the final url and only then append '.sig' suffix. This implies 2 things: - the signature payload initialization need to be moved to dload.c as it is the place where we have access to the resolved url - *.sig is downloaded serially with the main payload and this reduces level of parallelism Move *.sig payload creation to dload.c. Once the main payload is fetched successfully we check if the callee asked to download the accompanied signature. If yes - create a new payload and add it to mcurl. *.sig payload does not use server list of the main payload and thus does not support mirror failover. *.sig file comes from the same server as the main payload. Refactor event loop in curl_multi_download_internal() a bit. Instead of relying on curl_multi_check_finished_download() to return number of new payloads we simply rerun the loop iteration one more time to check if there are any active downloads left. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-06-26Cleanup the old sequential download codeAnatol Pomozov
All users of _alpm_download() have been refactored to the new API. It is time to remove the old _alpm_download() functionality now. This change also removes obsolete SIGPIPE signal handler functionality (this is a leftover from libfetch days). Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
2020-06-26Convert '-U pkg1 pkg2' codepath to parallel downloadAnatol Pomozov
Installing remote packages using its URL is an interesting case for ALPM API. Unlike package sync ('pacman -S pkg1 pkg2') '-U' does not deal with server mirror list. Thus _alpm_multi_download() should be able to handle file download for payloads that either have 'fileurl' field or pair of fields ('servers' and 'filepath') set. Signature for alpm_fetch_pkgurl() has changed and it accepts an output list that is populated with filepaths to fetched packages. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
2020-06-11build: add libintl dependency to meson and the .pc fileEli Schwartz
In order to use gettext on systems where it is not part of libc, the correct linker flags are needed in libalpm.pc (for static compilation). This has never been the case. The new meson build system currently only checks for ngettext in libc, but does not fall back to searching for the existence of -lintl; add it to the libalpm dependencies. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-06-01libalpm/signing.c: Fix calculation of packet size in parse_subpacketAllan McRae
Given RFC 4880 provides the code to do this calculation, I am not sure how I managed to stuff that up! This bug was only exposed when a signature made with "include-key-block" was added to the Arch repos, which provided a subpacket with the required size to hit this issue. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-06-01Remove autotools supportAllan McRae
This removes support for autotools in favour of meson.
2020-05-09Swap alpm_db_update() implementation to multiplexed versionAnatol Pomozov
Now when all callers of the old alpm_db_update() function are gone we can remove this implementation. And then rename alpm_dbs_update() function to alpm_db_update(). Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09Convert download packages logic to multiplexed APIAnatol Pomozov
Create a list of dload_payloads and pass it to the new _alpm_multi_* interface. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09Implement multibar UIAnatol Pomozov
Multiplexed download requires ability to draw UI for multiple active progress bars. To implement it we use ANSI codes to move cursor up/down and then redraw the required progress bar. `pacman_multibar_ui.active_downloads` field represents the list of active downloads that correspond to progress bars. `struct pacman_progress_bar` is a data structure for a progress bar. In some cases (e.g. database downloads) we want to keep progress bars in order. In some other cases (package downloads) we want to move completed items to the top of the screen. Function `multibar_move_completed_up` allows to configure such behavior. Per discussion in the maillist we do not want to show download progress for signature files. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09Extend download callback interface with start/complete eventsAnatol Pomozov
With the previous download interface the callback uses the first progress event as 'download has started' signal. Unfortunately it does not work with up-to-date files that never receive 'download progress' events. Up-to-date database messages are currently handled in sync_syncdbs() after the sequential download is completed and a result from ALPM is received. But this is not going to work with multiplexed download interface that returns the result only after all files are completed. Another problem with 'first progress event is the beginning of the download' is that such events time are unpredictable. Thus the UI progress bar order might differ from what has been passed by client to alpm_dbs_update() function. We actually want to keep the dbs progress bars in a strict order. To help to solve the given problems extend the download callback to allow 2 more events - download started and completed. 'Download started' events appear in the same order as in the list given by a client. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09Introduce event types for start/end database list downloadAnatol Pomozov
Multiplexed database/files downloads will use multiple progress bars. The UI logic is quite complicated and printing error messages while handling multiple progress bars is going to be challenging. Instead we are going to save all ALPM error messages to a list and flush it at the end of the download process. Use on_progress variable that blocks error messages printing. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09Implement multiplexed download using mCURLAnatol Pomozov
curl_multi_download_internal() is the main loop that creates up to 'ParallelDownloads' easy curl handles, adds them to mcurl and then performs curl execution. This is when the paralled downloads happens. Once any of the downloads complete the function checks its result. In case if the download fails it initiates retry with the next server from payload->servers list. At the download completion all the payload resources are cleaned up. curl_multi_check_finished_download() is essentially refactored version of curl_download_internal() adopted for multi_curl. Once mcurl porting is complete curl_download_internal() will be removed. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09Implement _alpm_multi_downloadAnatol Pomozov
It is an equivalent of _alpm_download but accepts a list of payloads. curl_multi_download_internal() is a stub at this moment and will be implemented in the later commits of this patch series. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09Inline dload_payload->curlerr field into a local variableAnatol Pomozov
dload_payload->curlerr is a field that is used inside curl_download_internal() function only. It can be converted to a local variable. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09Add multi_curl handle to ALPM global contextAnatol Pomozov
To be able to run multiple download in parallel efficiently we need to use curl_multi interface [1]. It introduces a set of APIs over new type of handler 'CURLM'. Create CURLM object at the application start and set it to global ALPM context. The 'single-download' CURL handle moves to payload struct. A new CURL handle is created for each payload with intention to be processed by CURLM. Note that curl_download_internal() is not ported to CURLM interface due to the fact that the function will go away soon. [1] https://curl.haxx.se/libcurl/c/libcurl-multi.html Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09Introduce alpm_dbs_update() function for parallel db updatesAnatol Pomozov
This is an equivalent of alpm_db_update but for multiplexed (parallel) download. The difference is that this function accepts list of databases to update. And then ALPM internals download it in parallel if possible. Add a stub for _alpm_multi_download the function that will do parallel payloads downloads in the future. Introduce dload_payload->filepath field that contains url path to the file we download. It is like fileurl field but does not contain protocol/server part. The rationale for having this field is that with the curl multidownload the server retry logic is going to move to a curl callback. And the callback needs to be able to reconstruct the 'next' fileurl. One will be able to do it by getting the next server url from 'servers' list and then concat with filepath. Once the 'parallel download' refactoring is over 'fileurl' field will go away. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09Add config option to specify amount of parallel download streamsAnatol Pomozov
It includes pacman.conf new 'ParallelDownloads' option that specifies how many concurrent downloads cURL starts in parallel. Add alpm_option_set_parallel_downloads() ALPM function that allows to set this config option programmatically. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-04-29Constify some input pointersRikard Falkeborn
Signed-off-by: Rikard Falkeborn <rikard.falkeborn@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-04-13Use GOTO_ERR throughoutAllan McRae
The GOTO_ERR define was added in commit 80ae8014 for use in future commits. There are plenty of places in the code base it can be used, so convert them. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-04-13Use STRDUP for error checking in more placesAllan McRae
Use STRDUP() over strdup() to catch memory allocation errors. There are still some instances of strdup left, but these are in functions that currently have no error path and would require a larger rework. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-04-13Add REALLOC macro to simplify realloc error handlingRikard Falkeborn
realloc can fail just like the other memory allocation functions. Add a macro to simplify handling of realloc failures, similar to the already existing MALLOC, CALLOC, etc. Replace the existing realloc uses with the new macro, allowing us to move tedious error handling to the macro. Also, in be_package and be_sync, this fixes hypothetical memory leaks (and thereafter null pointer dereferences) in case realloc fails to shrink the allocated memory. Signed-off-by: Rikard Falkeborn <rikard.falkeborn@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-03-13Add file and line number to RET_ERR{,_VOID}Allan McRae
Following the example of the recently added GOTO_ERR, adding the file and line number in addition to the function name in our debug messages is potentially useful. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-03-09Add GOTO_ERR() macro to set error and then goto a labelAnatol Pomozov
This is a macro similar to RET_ERR but useful in the case when we need to record an error and then jump to some cleanup section. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-03-09Remove unneeded goto cleanupAllan McRae
Since commit 2ee7a8d8, there is no cleanup needed in this function. Just return instead of jumping to the cleanup label. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-03-05Do not download files if find_dl_candidates() failsAnatol Pomozov
One reason why the function returns an error is some repo does not have any servers. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-02-24Simplify construction of payloads in download_filesAnatol Pomozov
Currently, download_files() creates payloads for all packages then iterates over them, calling download_single_file. This can be simplified by looping over packages and constructing the payload as needed. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-02-10build-aux/update-copyright 2019 2020Allan McRae
Signed-off-by: Allan McRae <allan@archlinux.org>
2020-01-31Eliminate extra loop over dbs_syncAnatol Pomozov
Current flow looks like loop dbs_sync { loop pkgs { if pkg.db == db then process(pkg, db) } } Package sync transaction always has a counterpart in the dbs_sync list (I cannot come up with a use-case when it is not true). So the loop can be simplified to: loop pkgs { process(pkg, pkg.db) } Tested: 'ninja test' & manually by using pacman with this patch for a week Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-01-28Docs docs docsmorganamilo
libalpm: move docs from .c files into alpm.h And fix/expand some along the way. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-01-28libalpm: fix alpm_option_set_assumeinstalledmorganamilo
It looks like this function has never actually worked. The current list is never set to NULL after being freed. So the new deps were just appended to the already freed list, leading to a segfault. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-01-27Fix "pacman -U <url>" operationsAllan McRae
Commit e6a6d307 detected complete part files by comparing a payload's max_size to initial_size. However, these values are also equal when we use pacman -U on a URL as max_size is set to 0 in that case. Add a further condition to avoid that. Signed-off-by: Allan McRae <allan@archlinux.org>