Code

V2#18

Code

Merged

breuleux merged 134 commits intomastermila-iqia/paperoni:masterfrom v2mila-iqia/paperoni:v2Copy head branch name to clipboardJan 10, 2023

+14,614-898

Commits

Commits on Oct 20, 2021

• Support search in Semantic Scholar (#6)

  Show description for 2c776a5
  notoraptor

  authored
  2c776a5

Commits on Oct 21, 2021

• Add SQL database schema, code and basic class (#7)

  Show description for 821e30c
  notoraptor

  authored
  821e30c

• Add option semantic-scholar -a <authorID> (#8)

  notoraptor

  authored
  a0c1dfb

Commits on Nov 9, 2021

• Add a command to import papers from JSON to a SQLite database. (#10)

  Show description for a7bce3a
  notoraptor

  authored
  a7bce3a

• Rework the data structures (#9)

  Show description for 36831f1
  breuleux

  authored
  36831f1

Commits on Nov 26, 2021

• Directly collect Semantic Scholar papers into SQL database (#11)

  Show description for 220bd30
  notoraptor

  authored
  220bd30

Commits on Jan 18, 2022

• Add command to search into SQL database.

  Show description for a8b8a3c
  

  notoraptor

  authored and

  breuleux

  committed
  a8b8a3c

• Reorganize code.

  Show description for 425924e
  

  notoraptor

  authored and

  breuleux

  committed
  425924e

• Correctly download PDF from papers2.Paper class.

  

  notoraptor

  authored and

  breuleux

  committed
  450493a

• Catch MutuallyExclusiveError error at top-level in command_search2 and print a proper error message.

  

  notoraptor

  authored and

  breuleux

  committed
  17490e5

• Make sure to return boolean value where Database.select_id_from_values() is used to find a paper.

  

  notoraptor

  authored and

  breuleux

  committed
  40de8a6

• Add a static list of source types to use to get paper and author identifiers.

  

  notoraptor

  authored and

  breuleux

  committed
  f1c2816

• Add a database example by importing mila-papers.json.

  Show description for e40a214
  

  notoraptor

  authored and

  breuleux

  committed
  e40a214

• [tests]

  Show description for 0a8e287
  

  notoraptor

  authored and

  breuleux

  committed
  0a8e287

• Update docstring for command_search2

  Show description for 1367170
  

  notoraptor

  and

  breuleux

  committed
  1367170

Commits on Feb 1, 2022

• Add search command for OpenAlex.

  

  notoraptor

  authored and

  breuleux

  committed
  27b9423

• Fix typo

  Show description for 634a7dd
  

  notoraptor

  and

  breuleux

  committed
  634a7dd

Commits on Jun 22, 2022

• Temporarily set old paperoni code aside

  breuleux

  committed
  eb38f23

• Move sql schema

  breuleux

  committed
  8c034a0

Commits on Jul 4, 2022

• Unpin dev versions

  breuleux

  committed
  060a621

• Add affiliation table and paper flags

  breuleux

  committed
  01facca

Commits on Jul 13, 2022

• Add automatic schema

  breuleux

  committed
  f4b6f0b

Commits on Jul 19, 2022

• Update database and regenerate schema

  breuleux

  committed
  2eb2792

• Add Pydantic model

  breuleux

  committed
  9276499

• Add script to convert existing MAG data

  breuleux

  committed
  f0504e2

• Add utils

  breuleux

  committed
  cd1b9c8

• Add paper_scraper table

  breuleux

  committed
  b5b359c

• Update conversion scripts

  breuleux

  committed
  9b90d53

• Update database schema

  breuleux

  committed
  62b72cf

• Adjustments to the model

  breuleux

  committed
  18af34a

• Add db autocreation/import, semantic scholar

  breuleux

  committed
  a859f3b

• Improve history and replay mechanism

  breuleux

  committed
  473d154

• Add primary keys to certain tables

  breuleux

  committed
  219ae6d

• Define display using ovld

  breuleux

  committed
  13c9a07

• Expand configuration code

  breuleux

  committed
  dab3ba8

Commits on Jul 20, 2022

• Finalize author acquisition code for semantic scholar

  breuleux

  committed
  7a7b416

• Commit config.yaml file

  breuleux

  committed
  c695720

• Avoid accumulating bad aliases, display aliases/years

  breuleux

  committed
  6fa731f

• Improve prepare command

  breuleux

  committed
  c88b2fd

Commits on Jul 21, 2022

• Change notation for noids

  breuleux

  committed
  7ec2a8d

• Define acquire for semantic scholar

  breuleux

  committed
  cea2fc5

• Better way to find and load scrapers

  breuleux

  committed
  63a7243

• Add documentation to add a new scraper

  breuleux

  committed
  cf1aa0a

• Fix bug, remove name field in scraper class

  breuleux

  committed
  34234f2

Commits on Jul 29, 2022

• Add OpenReview scraper query method (#15)

  satyaog

  authored
  2856d8c

• Use hash-based id (step 1)

  breuleux

  committed
  4310113

• Use hash-based id (step 2)

  breuleux

  committed
  3481d96

• Handle None hashids

  breuleux

  committed
  62fe9a0

• Remove AuthorQuery, add UniqueAuthor

  breuleux

  committed
  1eb526c

• Move uuid_tag stuff to tools.py

  breuleux

  committed
  50c7c3b

• Move affiliations to PaperAuthor, acquire returns ids

  breuleux

  committed
  dd936a0

• Add and use canonical_id table

  breuleux

  committed
  bbeea4d

• Add merging logic

  breuleux

  committed
  db7f0c5

• Adjust openreview scraper for model modification

  breuleux

  committed
  e4ac4df

Commits on Aug 1, 2022

• Fix authorids usage in OpenReview scraper (#16)

  Show description for 020b86b
  satyaog

  authored
  020b86b

• Add OpenReview scraper acquire method (#17)

  satyaog

  authored
  e0f8c7f

• Add search, fix some bugs with merging

  breuleux

  committed
  3b1bd42

• Use ovld instead of match statement in database.py

  breuleux

  committed
  fa9f699

• Make merge more general, log to history

  breuleux

  committed
  4886391

• Make canonical ID deterministic for replay

  breuleux

  committed
  2d0b2df

• Add sql command, search by author

  breuleux

  committed
  48ed967

• Add more search options and reports

  breuleux

  committed
  41ebbfb

• Add rich dependency

  breuleux

  committed
  193be4d

• fixup! Add more search options and reports

  breuleux

  committed
  4ee14ad

• Add sources.utils.prepare

  breuleux

  committed
  2dc0166

• Minor adjustments to openreview scraper

  breuleux

  committed
  3472143

Commits on Aug 2, 2022

• Add prepare method to openreview scraper

  breuleux

  committed
  e2472e6

Commits on Aug 3, 2022

• Simplify paper_author_institution table, fix merge bug

  breuleux

  committed
  ff9d002

• Change scraper table to be type agnostic

  breuleux

  committed
  cc01ceb

• Include principal name in aliases table

  breuleux

  committed
  bb39f26

• Remove year from venue name in OpenReview scraper

  breuleux

  committed
  f4f9f7e

Commits on Aug 4, 2022

• Allow fetching from all venues in prepare

  breuleux

  committed
  d849a04

Commits on Aug 10, 2022

• Redesign release/venue and note scraper metainfo differently

  breuleux

  committed
  106cfdd

Commits on Aug 11, 2022

• Scrape venues from OpenReview

  breuleux

  committed
  abb97a6

Commits on Aug 12, 2022

• Move .sources.model to .model

  breuleux

  committed
  6d6cfa1

• Improve software architecture for merging

  breuleux

  committed
  24f5474

• Add more author merging methods

  breuleux

  committed
  f4e36c1

Commits on Aug 15, 2022

• Add OpenReview venues scraper and venue merging

  breuleux

  committed
  7c5c280

Commits on Aug 16, 2022

• Integrate a quality measure to paper/author/venue for better merging

  breuleux

  committed
  356aa13

• Add _process_standard_rows

  breuleux

  committed
  d02904a

• Fix MAG ids for venues

  breuleux

  committed
  bd2bcd2

Commits on Aug 17, 2022

• Misc improvements

  breuleux

  committed
  4c01165

Commits on Aug 19, 2022

• Improvements in data handling

  Show description for 86263b5
  breuleux

  committed
  86263b5

• Refactor utils and tools into tools and display

  breuleux

  committed
  c7e2ca7

• Add HTML display, fix date wrangling

  breuleux

  committed
  cef14fc

Commits on Aug 31, 2022

• Properly set aliases for institutions and venues

  breuleux

  committed
  55f9039

• Add more possible date formats

  breuleux

  committed
  ba7ea06

• Show citation_count conditionally

  breuleux

  committed
  5de00ed

Commits on Sep 7, 2022

• Move load_config/database to config.py

  breuleux

  committed
  2719772

• Add paper refinement methods

  breuleux

  committed
  ed79b1e

Commits on Sep 9, 2022

• Modifications to scraper interface

  breuleux

  committed
  77ae6be

Commits on Sep 12, 2022

• Add factory to make models with ids attached

  breuleux

  committed
  f7addb6

Commits on Sep 14, 2022

• Move old tests out of the way for now

  breuleux

  committed
  f781d8c

• Improve affiliation extraction

  breuleux

  committed
  e4d73da

• Test tools.py

  breuleux

  committed
  09fde51

• Simple test for replay feature

  breuleux

  committed
  7e1914a

Commits on Sep 16, 2022

• Add more refinement methods

  breuleux

  committed
  98ff4f5

• Improve affiliation acquisition from pdfs

  breuleux

  committed
  298e8f6

Commits on Sep 20, 2022

• Disable citations fields in semantic scholar

  breuleux

  committed
  42af254

Commits on Sep 21, 2022

• Commit default.html for HTML production

  breuleux

  committed
  c8dff39

• Use readpage() for HTTPSAcquirer

  breuleux

  committed
  f4cee75

• Test Semantic Scholar query()

  breuleux

  committed
  339e66b

• Misc improvements, improve affiliations from PDFs

  breuleux

  committed
  e65e4d9

• Use requests-cache

  breuleux

  committed
  28b4b74

Commits on Sep 22, 2022

• Improve configuration management

  breuleux

  committed
  2714c78

Commits on Sep 23, 2022

• Scrape affiliations from ScienceDirect

  breuleux

  committed
  91ebc21

Commits on Sep 26, 2022

• Config processing fixes, improvements to refine

  breuleux

  committed
  f9b87ac

Commits on Oct 4, 2022

• Misc improvements

  breuleux

  committed
  da4668e

Commits on Oct 26, 2022

• Test model.py with full coverage

  breuleux

  committed
  9d2c001

• Improvements in tests and test dbs

  breuleux

  committed
  c21f463

• Add utilities for scraper tests

  breuleux

  committed
  9abc547

Commits on Oct 28, 2022

• Simplify and test scrapers/base

  breuleux

  committed
  04bdeab

• Fully test semantic_scholar scraper

  breuleux

  committed
  f4c2ec6

Commits on Oct 29, 2022

• Test openreview scraper

  breuleux

  committed
  261bf6a

Commits on Nov 2, 2022

• Add make_date

  breuleux

  committed
  fe01f3c

• Implement covguard to help with coverage

  breuleux

  committed
  f5fc6bb

• Add openreview.get_profile, better tests and coverage

  breuleux

  committed
  47b7977

Commits on Nov 3, 2022

• Use pytest-regressions package

  breuleux

  committed
  60a99da

• Improve and test refine

  breuleux

  committed
  b9e84d8

• Test display.py

  breuleux

  committed
  d9a08a0

• Add out_regression fixture

  breuleux

  committed
  638a139

Commits on Jan 6, 2023

• Test some edge cases

  breuleux

  committed
  9b5ebaf

• Add ScraperData

  breuleux

  committed
  364ebe8

• Remove screenshot from old paperoni

  breuleux

  committed
  29f67c2

• Modify cache location for tests, regularize empty db

  breuleux

  committed
  ec0d7c2

Commits on Jan 7, 2023

• Add script to regenerate the schema

  breuleux

  committed
  972561b

Commits on Jan 9, 2023

• Update dependencies

  breuleux

  committed
  7a356ca

• Refactor formatters in cli

  breuleux

  committed
  aced2db

• Write new-coverage files when non-covered code is triggered

  breuleux

  committed
  9447c24

• OpenReviewProfileScraper

  breuleux

  committed
  4e5c291

• Add unaccented location names to pdftools

  breuleux

  committed
  1dc7acd

• Make default config simpler

  breuleux

  committed
  1ac35fa

Commits on Jan 10, 2023

• Add query part of zeta-alpha scraper

  breuleux

  committed
  70abd1d

• Add small explanatory blurb in README

  breuleux

  committed
  3dfac28