Skip to content

Commit

Permalink
#517 - update nexB/purldb to aboutcode-org/purldb
Browse files Browse the repository at this point in the history
Signed-off-by: Chin Yeung Li <[email protected]>
  • Loading branch information
chinyeungli committed Aug 9, 2024
1 parent 02389be commit 6018577
Show file tree
Hide file tree
Showing 209 changed files with 2,883 additions and 1,607 deletions.
16 changes: 8 additions & 8 deletions CHANGELOG.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,21 +4,21 @@ Changelog
Next Release
----------------

- Improve web template for API web page https://github.com/nexB/purldb/issues/132
- Improve web template for API web page https://github.com/aboutcode-org/purldb/issues/132
- The API endpoints for ``approximate_directory_content_index``,
``approximate_directory_structure_index``, ``exact_file_index``,
``exact_package_archive_index``, ``cditems``, ``on_demand_queue`` have been
removed.
- The `/api/collect/` and `/api/collect/index_packages/` API endpoints have been
updated such that Package scan and processing requests made with purls with
versions are processed ahead of those made with versionless purls.
https://github.com/nexB/purldb/issues/502
https://github.com/aboutcode-org/purldb/issues/502
- The `/api/scan_queue/` endpoint has been updated.
`/api/scan_queue/get_next_download_url/` now returns a `webhook_url`, where
the purldb scan worker will submit results. This is a URL to the
`/api/scan_queue/index_package_scan/` endpoint.
`/api/scan_queue/update_status/` is now an action on a ScannableURI.
https://github.com/nexB/purldb/issues/504
https://github.com/aboutcode-org/purldb/issues/504


v5.0.0
Expand All @@ -27,11 +27,11 @@ v5.0.0
- Add `/api/docs` Swagger API documentation for API endpoints.
- Add `/api/from_purl/purl2git` endpoint to get a git repo for a purl.
- Add `/api/to_purl/go` endpoint to get a purl from a golang import string or a package string in go.mod.
- Support indexing of PURLs listed in https://github.com/nexB/purldb/issues/326,
https://github.com/nexB/purldb/issues/327, https://github.com/nexB/purldb/issues/328,
https://github.com/nexB/purldb/issues/329 and https://github.com/nexB/purldb/issues/356.
- Support ``addon_pipelines`` for symbol and string collection in ``/api/collect`` endpoint. https://github.com/nexB/purldb/pull/393
- Store ``source_symbols`` and ``source_strings`` in ``extra_data`` field. https://github.com/nexB/purldb/pull/351
- Support indexing of PURLs listed in https://github.com/aboutcode-org/purldb/issues/326,
https://github.com/aboutcode-org/purldb/issues/327, https://github.com/aboutcode-org/purldb/issues/328,
https://github.com/aboutcode-org/purldb/issues/329 and https://github.com/aboutcode-org/purldb/issues/356.
- Support ``addon_pipelines`` for symbol and string collection in ``/api/collect`` endpoint. https://github.com/aboutcode-org/purldb/pull/393
- Store ``source_symbols`` and ``source_strings`` in ``extra_data`` field. https://github.com/aboutcode-org/purldb/pull/351


v3.0.0
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

FROM --platform=linux/amd64 python:3.12-slim

LABEL org.opencontainers.image.source="https://github.com/nexB/purldb"
LABEL org.opencontainers.image.source="https://github.com/aboutcode-org/purldb"
LABEL org.opencontainers.image.description="PurlDB"
LABEL org.opencontainers.image.licenses="Apache-2.0"

Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# purldb is a trademark of nexB Inc.
# SPDX-License-Identifier: Apache-2.0
# See http://www.apache.org/licenses/LICENSE-2.0 for the license text.
# See https://github.com/nexB/purldb for support or download.
# See https://github.com/aboutcode-org/purldb for support or download.
# See https://aboutcode.org for more information about nexB OSS projects.
#

Expand Down
2 changes: 1 addition & 1 deletion NOTICE
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,6 @@
# See https://www.apache.org/licenses/LICENSE-2.0 for the license text.
# See https://creativecommons.org/licenses/by-sa/4.0/legalcode for the license text.
#
# See https://github.com/nexB/purldb for support or download.
# See https://github.com/aboutcode-org/purldb for support or download.
# See https://aboutcode.org for more information about nexB OSS projects.
#
2 changes: 1 addition & 1 deletion clearcode/management/commands/clearload.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# purldb is a trademark of nexB Inc.
# SPDX-License-Identifier: Apache-2.0
# See http://www.apache.org/licenses/LICENSE-2.0 for the license text.
# See https://github.com/nexB/purldb for support or download.
# See https://github.com/aboutcode-org/purldb for support or download.
# See https://aboutcode.org for more information about nexB OSS projects.
#

Expand Down
2 changes: 1 addition & 1 deletion clearcode/management/commands/clearsync.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# purldb is a trademark of nexB Inc.
# SPDX-License-Identifier: Apache-2.0
# See http://www.apache.org/licenses/LICENSE-2.0 for the license text.
# See https://github.com/nexB/purldb for support or download.
# See https://github.com/aboutcode-org/purldb for support or download.
# See https://aboutcode.org for more information about nexB OSS projects.
#

Expand Down
6 changes: 4 additions & 2 deletions clearcode/management/commands/store_scans.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,14 @@
# purldb is a trademark of nexB Inc.
# SPDX-License-Identifier: Apache-2.0
# See http://www.apache.org/licenses/LICENSE-2.0 for the license text.
# See https://github.com/nexB/purldb for support or download.
# See https://github.com/aboutcode-org/purldb for support or download.
# See https://aboutcode.org for more information about nexB OSS projects.
#

from clearcode.store_scans import store_scancode_scans_from_cd_items
from minecode.management.commands import VerboseCommand


class Command(VerboseCommand):
help = 'Store scancode scans in git repositories'

Expand All @@ -19,4 +20,5 @@ def add_arguments(self, parser):
parser.add_argument('--count', type=int, default=0)

def handle(self, *args, **options):
store_scancode_scans_from_cd_items(work_dir=options['work_dir'], github_org=options['github_org'], count=options['count'])
store_scancode_scans_from_cd_items(
work_dir=options['work_dir'], github_org=options['github_org'], count=options['count'])
32 changes: 21 additions & 11 deletions clearindex/harvest.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# purldb is a trademark of nexB Inc.
# SPDX-License-Identifier: Apache-2.0
# See http://www.apache.org/licenses/LICENSE-2.0 for the license text.
# See https://github.com/nexB/purldb for support or download.
# See https://github.com/aboutcode-org/purldb for support or download.
# See https://aboutcode.org for more information about nexB OSS projects.
#

Expand Down Expand Up @@ -85,7 +85,8 @@ def create_from_harvest(package_scan={}, files_data=[], cditem_path=''):
'source_packages',
)

package_data = {field_name: package_scan.get(field_name) for field_name in fields}
package_data = {field_name: package_scan.get(
field_name) for field_name in fields}

stringify_null_purl_fields(package_data)

Expand All @@ -98,7 +99,8 @@ def create_from_harvest(package_scan={}, files_data=[], cditem_path=''):

download_url = package_data.get('download_url')
if not download_url:
logger.error('Null `download_url` value for `package_data`: {}'.format(package_data))
logger.error(
'Null `download_url` value for `package_data`: {}'.format(package_data))
return

# This ugly block is needed until https://github.com/nexB/packagedb/issues/14
Expand All @@ -119,9 +121,11 @@ def create_from_harvest(package_scan={}, files_data=[], cditem_path=''):
new_package_data=package_data,
replace=False
)
package.append_to_history('Updated package from CDitem harvest: {}'.format(cditem_path))
package.append_to_history(
'Updated package from CDitem harvest: {}'.format(cditem_path))

logger.info('Merged package data from scancode harvest: {}'.format(package))
logger.info(
'Merged package data from scancode harvest: {}'.format(package))

except Package.DoesNotExist:
try:
Expand All @@ -132,15 +136,19 @@ def create_from_harvest(package_scan={}, files_data=[], cditem_path=''):
new_package_data=package_data,
replace=False
)
package.append_to_history('Updated package from CDitem harvest: {}'.format(cditem_path))
package.append_to_history(
'Updated package from CDitem harvest: {}'.format(cditem_path))

logger.info('Merged package data from scancode harvest: {}'.format(package))
logger.info(
'Merged package data from scancode harvest: {}'.format(package))

except Package.DoesNotExist:
package = Package.objects.create(**package_data)
package.append_to_history('Created package from CDitem harvest: {}'.format(cditem_path))
package.append_to_history(
'Created package from CDitem harvest: {}'.format(cditem_path))

logger.info('Created package from scancode harvest: {}'.format(package))
logger.info(
'Created package from scancode harvest: {}'.format(package))

# Now, add resources to the Package.
for f in files_data:
Expand Down Expand Up @@ -179,7 +187,8 @@ def map_scancode_harvest(cditem):
try:
harvest_data = cditem.data
except ValueError:
err_msg = 'CDitemError: empty content field for CDitem: {}'.format(cditem.path)
err_msg = 'CDitemError: empty content field for CDitem: {}'.format(
cditem.path)
logger.error(err_msg)

cditem.map_error = err_msg
Expand All @@ -198,7 +207,8 @@ def map_scancode_harvest(cditem):
download_url = package_scan.get('download_url')
if not download_url:
purl = package_scan.get('purl')
err_msg = 'CDitemError: empty download_url for package: {}'.format(purl)
err_msg = 'CDitemError: empty download_url for package: {}'.format(
purl)
logger.error(err_msg)

cditem.map_error = err_msg
Expand Down
28 changes: 18 additions & 10 deletions clearindex/management/commands/run_clearindex.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# purldb is a trademark of nexB Inc.
# SPDX-License-Identifier: Apache-2.0
# See http://www.apache.org/licenses/LICENSE-2.0 for the license text.
# See https://github.com/nexB/purldb for support or download.
# See https://github.com/aboutcode-org/purldb for support or download.
# See https://aboutcode.org for more information about nexB OSS projects.
#

Expand Down Expand Up @@ -118,8 +118,10 @@ def handle(self, *args, **options):
logger.info('Graceful exit of the map loop.')
break

mappable_definitions = CDitem.objects.mappable_definitions()[:MAP_BATCH_SIZE]
mappable_scancode_harvests = CDitem.objects.mappable_scancode_harvests()[:MAP_BATCH_SIZE]
mappable_definitions = CDitem.objects.mappable_definitions()[
:MAP_BATCH_SIZE]
mappable_scancode_harvests = CDitem.objects.mappable_scancode_harvests()[
:MAP_BATCH_SIZE]

try:
if not mappable_definitions and not mappable_scancode_harvests:
Expand Down Expand Up @@ -187,7 +189,7 @@ def get_coords_des_and_lic_from_def(definition):
return definition.get('coordinates', {}), definition.get('described', {}), definition.get('licensed', {})


#CD_TYPES_WITH_SOURCE = ('debsrc', 'npm', 'sourcearchive',)
# CD_TYPES_WITH_SOURCE = ('debsrc', 'npm', 'sourcearchive',)


def get_or_create_package_from_cditem_definition(cditem):
Expand All @@ -197,7 +199,8 @@ def get_or_create_package_from_cditem_definition(cditem):
definition = cditem.data
if not definition:
raise Exception('No data available for this definition')
coordinates, described, licensed = get_coords_des_and_lic_from_def(definition)
coordinates, described, licensed = get_coords_des_and_lic_from_def(
definition)

download_url = described.get('urls', {}).get('download', '')
if not download_url:
Expand All @@ -219,15 +222,17 @@ def get_or_create_package_from_cditem_definition(cditem):
name = coordinates.get('name')
version = coordinates.get('revision')
package_type = coordinates.get('type')
converted_package_type = PACKAGE_TYPES_BY_CD_TYPE.get(package_type) or package_type
converted_package_type = PACKAGE_TYPES_BY_CD_TYPE.get(
package_type) or package_type
# TODO: Source packages need to be updated for clearlydefined, link source packages to binary packages
hashes = described.get('hashes', {})
sha1 = hashes.get('sha1')
sha256 = hashes.get('sha256')
homepage_url = described.get('projectWebsite')
release_date = described.get('releaseDate')
declared_license = licensed.get('declared')
normalized_license_expression = licensing.get_normalized_expression(declared_license)
normalized_license_expression = licensing.get_normalized_expression(
declared_license)
copyrights = get_parties_from_licensed(licensed)
copyrights = '\n'.join(copyrights)
definition_mining_level = 0
Expand Down Expand Up @@ -257,7 +262,8 @@ def get_or_create_package_from_cditem_definition(cditem):
)
# log history if package was created
if created:
package.append_to_history('Created package from CDitem definition: {}'.format(cditem.path))
package.append_to_history(
'Created package from CDitem definition: {}'.format(cditem.path))

else:
# TODO: This is temporary until we fold clearindex into minecode mapping
Expand All @@ -283,7 +289,8 @@ def get_or_create_package_from_cditem_definition(cditem):
replace=True
)
package = existing_package
package.append_to_history('Updated package from CDitem definition: {}'.format(cditem.path))
package.append_to_history(
'Updated package from CDitem definition: {}'.format(cditem.path))

return package

Expand All @@ -303,7 +310,8 @@ def create_download_url_from_coords(coord):

package_type = PACKAGE_TYPES_BY_CD_TYPE.get(ptype)
if not package_type:
raise Exception('Unsupported ClearlyDefined package type: {}'.format(ptype))
raise Exception(
'Unsupported ClearlyDefined package type: {}'.format(ptype))

get_urls = PACKAGE_TYPES_WITH_GET_URLS.get(package_type)
if get_urls:
Expand Down
2 changes: 1 addition & 1 deletion clearindex/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# purldb is a trademark of nexB Inc.
# SPDX-License-Identifier: Apache-2.0
# See http://www.apache.org/licenses/LICENSE-2.0 for the license text.
# See https://github.com/nexB/purldb for support or download.
# See https://github.com/aboutcode-org/purldb for support or download.
# See https://aboutcode.org for more information about nexB OSS projects.
#

Expand Down
18 changes: 9 additions & 9 deletions docs/source/contributing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ Do Your Homework
----------------

Before adding a contribution or create a new issue, take a look at the project’s
`README <https://github.com/nexB/purldb#readme>`_, read through our
`README <https://github.com/aboutcode-org/purldb#readme>`_, read through our
`documentation <https://purldb.readthedocs.io>`_,
and browse existing `issues <https://github.com/nexB/purldb/issues>`_,
and browse existing `issues <https://github.com/aboutcode-org/purldb/issues>`_,
to develop some understanding of the project and confirm whether a given
issue/feature has previously been discussed.

Expand All @@ -34,7 +34,7 @@ First Timers

You are here to help, but you're a new contributor! No worries, we always
welcome newcomer contributors. We maintain some
`good first issues <https://github.com/nexB/purldb/labels/good%20first%20issue>`_
`good first issues <https://github.com/aboutcode-org/purldb/labels/good%20first%20issue>`_
and encourage new contributors to work on those issues for a smooth start.

.. warning::
Expand All @@ -55,15 +55,15 @@ and encourage new contributors to work on those issues for a smooth start.
Report Issues
^^^^^^^^^^^^^

- Report a new `bug <https://github.com/nexB/purldb/issues>`_; just remember to be
- Report a new `bug <https://github.com/aboutcode-org/purldb/issues>`_; just remember to be
as specific as possible.
- Create a `new issue <https://github.com/nexB/purldb/issues>`_ to request a
- Create a `new issue <https://github.com/aboutcode-org/purldb/issues>`_ to request a
feature, submit a feedback, or ask a question.
- Look into existing `bugs <https://github.com/nexB/purldb/labels/bug>`_,
- Look into existing `bugs <https://github.com/aboutcode-org/purldb/labels/bug>`_,
try to reproduce the issue on your side, and discuss solutions in the comments.

.. note::
Make sure to check existing `issues <https://github.com/nexB/purldb/issues>`_,
Make sure to check existing `issues <https://github.com/aboutcode-org/purldb/issues>`_,
and :ref:`faq` to confirm whether a given issue or a question has previously been
discussed.

Expand All @@ -72,7 +72,7 @@ Code Contributions

Code is contributed to the codebase using **pull requests**.
A pull request should always be attached to an existing issue.
When there is no existing issues, start by `creating one <https://github.com/nexB/purldb/issues>`_
When there is no existing issues, start by `creating one <https://github.com/aboutcode-org/purldb/issues>`_
to discuss potential solutions and implementation details before writing any code.

We use several conventions to ensure code quality regarding format, testing, and
Expand Down Expand Up @@ -102,7 +102,7 @@ See also :ref:`testing` for module-specific testing commands.
3. **Commit messages and Developer Certificate of Origin**

Follow the instructions at `Writing good Commit Messages <https://aboutcode.readthedocs.io/en/latest/contributing/writing_good_commit_messages.html>`_
and `check some examples <https://github.com/nexB/purldb/commits/main>`_.
and `check some examples <https://github.com/aboutcode-org/purldb/commits/main>`_.

**You must include a "Signed-off-by" to your commit messages**::

Expand Down
2 changes: 1 addition & 1 deletion docs/source/how-to-guides/deploy_to_devel.rst
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ Install PurlDB and matchcode.io from a clone of the PurlDB git repository

Then run these commands::

git clone https://github.com/nexB/purldb
git clone https://github.com/aboutcode-org/purldb
cd purldb
make dev
make envfile
Expand Down
4 changes: 2 additions & 2 deletions docs/source/how-to-guides/installation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -178,9 +178,9 @@ latest/getting-started/install.html#prerequisites>`_ for more details.
Clone and Configure
^^^^^^^^^^^^^^^^^^^

* Clone the `PurlDB GitHub repository <https://github.com/nexB/purldb>`_::
* Clone the `PurlDB GitHub repository <https://github.com/aboutcode-org/purldb>`_::

git clone https://github.com/nexB/purldb.git && cd purldb
git clone https://github.com/aboutcode-org/purldb.git && cd purldb

* Inside the :guilabel:`purldb/` directory, install the required dependencies::

Expand Down
2 changes: 1 addition & 1 deletion docs/source/license.rst
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,6 @@ See https://www.apache.org/licenses/LICENSE-2.0 for the license text.

See https://creativecommons.org/licenses/by-sa/4.0/legalcode for the license text.

See https://github.com/nexB/purldb for support or download.
See https://github.com/aboutcode-org/purldb for support or download.

See https://aboutcode.org for more information about nexB OSS projects.
2 changes: 1 addition & 1 deletion docs/source/purldb/rest_api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1283,7 +1283,7 @@ matching
Given a ScanCode.io JSON output ``upload_file``, match directory and resources
of the codebase in ``upload_file`` to Packages indexed in the PurlDB.
This endpoint runs the ``matching`` pipeline at https://github.com/nexB/purldb/blob/main/matchcode_pipeline/pipelines/matching.py
This endpoint runs the ``matching`` pipeline at https://github.com/aboutcode-org/purldb/blob/main/matchcode_pipeline/pipelines/matching.py
Using cURL to upload a scan for matching:
Expand Down
Loading

0 comments on commit 6018577

Please sign in to comment.