This repository contains parallel language corpus links for popular Indian languages developed as part of the Anuvaad project.
Please reach out to [email protected] for any clarification/interpretation/usage of the linked datasets.
Shield:
This work is licensed under a Creative Commons Attribution 4.0 International License.
The current status of the parallel corpus built :
(Going forward, Anuvaad data contribution will be done via ULCA platform. Please refer https://bhashini.gov.in/ulca for further details)
| Language Pair | Parallel Corpus Count |
|---|---|
| English-Hindi | 5,627,366 |
| English-Bengali | 1,633,224 |
| English-Tamil | 1,522,854 |
| English-Malayalam | 1,505,760 |
| English-Telugu | 1,660,974 |
| English-Kannada | 1,352,017 |
| English-Marathi | 2,537,853 |
| English-Punjabi | 2,154,298 |
| English-Gujarati | 1,656,616 |
| English-Assamese | 47,174 |
| English-Urdu | 162,048 |
| English-Odia | 92,309 |
The goal is to build high quality parallel corpus for the Indian languages across various domains (General, Legal, Education, Healthcare, Automobile, News etc). This can be eventually used to train the ML models based on the use cases.
Read more about Anuvaad @ http://anuvaad.org/
The code for building the below mentioned datasets are available under https://github.com/project-anuvaad/anuvaad-corpus-tools
| Domain | Source | Year | En-Hi pairs count | Corpus Download Link |
|---|---|---|---|---|
| Legal | Multiple Sources | 2010-2020 | 947,148 | Indian Judiciary |
| Legal | HC/SUVAS | NA | 81,884 | HC/SUVAS |
| Legal | Legal Terminologies | NA | 3,980 | Legal Terms |
| Legal | Nouns Dictionary | NA | 72,192 | Nouns Dict |
| News | PIB | 2017-2020 | 401,859 | PIB (2017-2020) |
| News | PIB Archives | 2009-2016 | 288,573 | PIB Archives (2009-2016) |
| News | NewsOnAir | 2019-2021(Jan) | 110,748 | NewsOnAir (2019-2021) |
| General | Wikipedia | Until 2021-02 | 193,340 | Wikipedia (20210201) |
| Automobile | DriveSpark | Until 2021-03 | 39,545 | DriveSpark (20210303) |
| General | DD News National | Until 2021-03 | 33,263 | DD News-National (20210320) |
| Sports | DD News Sports | Until 2021-03 | 3,666 | DD News-Sports (20210320) |
| Tourism | Native Planet | Until 2021-03 | 31,933 | Native Planet (20210320) |
| News | CatchNews | Until 2021-03 | 35,525 | CatchNews (20210320) |
| News | DW News | 2008-2020 | 68,875 | DW News (2008-2020) |
| News | OneIndia | Until 2021-03 | 143,139 | OneIndia (20210320) |
| Sports | MyKhel | Until 2021-03 | 15,711 | MyKhel (20210320) |
| Financial | GoodReturns | Until 2021-03 | 23,587 | GoodReturns (20210320) |
| Sports | Indian Express | Until 2021-03 | 74,265 | IE-Sports (20210320) |
| Technology | Indian Express | Until 2021-03 | 57,625 | IE-Technology (20210320) |
| News | Indian Express | Until 2021-03 | 170,917 | IE-News (20210320) |
| Lifestyle | Indian Express | Until 2021-03 | 1,812 | IE-Lifestyle (20210320) |
| General | Indian Express | Until 2021-03 | 547 | IE-General (20210320) |
| Entertainment | Indian Express | Until 2021-03 | 43,202 | IE-Entertainment (20210320) |
| Education | Indian Express | Until 2021-03 | 18,898 | IE-Education (20210320) |
| Financial | Indian Express | Until 2021-03 | 16,212 | IE-Financial (20210320) |
| News | Times of India | Until 2021-03 | 653,356 | TOI-News (20210320) |
| Financial | Financial Express | Until 2021-03 | 13,345 | Financial Express (20210320) |
| News | The Wire | Until 2021-03 | 98,137 | The Wire (20210320) |
| News | Tribune | Until 2021-03 | 391,004 | Tribune (20210320) |
| Financial | ZeeBiz | Until 2021-03 | 4,338 | ZeeBiz (20210320) |
| Government Press Release | Punjab Govt | As of 2021-03 | 12,106 | Punjab Govt (20210320) |
| General | The Better India | Until 2021-03 | 6,217 | The Better India (20210320) |
| News | Jagran | Until 2021-03 | 75,002 | Jagran-News (20210320) |
| Technology | Jagran | Until 2021-03 | 63,555 | Jagran-Technology (20210320) |
| Education | Jagran | Until 2021-03 | 26,770 | Jagran-Education (20210320) |
| Entertainment | Jagran | Until 2021-03 | 50,328 | Jagran-Entertainment (20210320) |
| Financial | Jagran | Until 2021-03 | 26,497 | Jagran-Financial (20210320) |
| Sports | Jagran | Until 2021-03 | 34,190 | Jagran-Sports (20210320) |
| Lifestyle | Jagran | Until 2021-03 | 6,253 | Jagran-Lifestyle (20210320) |
| News | AsianetNews | Until 2021-03 | 15,087 | AsianetNews (20210320) |
| Financial | Business Standard | Until 2021-03 | 268,603 | Business Standard (20210320) |
| Government Press Release | Pranabmukherjee | As of 2021-03 | 43,259 | Pranabmukherjee (20210320) |
| Entertainment | Lokmat | Until 2021-05 | 16,873 | Lokmat-Entertainment (20210501) |
| News | Lokmat | Until 2021-05 | 68,793 | Lokmat-News (20210501) |
| Lifestyle | Lokmat | Until 2021-05 | 1,223 | Lokmat-Lifestyle (20210501) |
| Sports | Lokmat | Until 2021-05 | 36,279 | Lokmat-Sports (20210501) |
| Technology | Lokmat | Until 2021-05 | 1,946 | Lokmat-Technology (20210501) |
| Financial | Lokmat | Until 2021-05 | 6,751 | Lokmat-Financial (20210501) |
| Healthcare | Lokmat | Until 2021-05 | 2,202 | Lokmat-Healthcare (20210501) |
| News | Zee News | Until 2021-04 | 870,065 | Zee-News (2011-2021) |
| Domain | Source | Year | En-Bn pairs count | Corpus Download Link |
|---|---|---|---|---|
| Legal | Multiple Sources | 2010-2020 | 625,456 | Indian Judiciary |
| Legal | HC/SUVAS | NA | 18,353 | HC/SUVAS |
| News | PIB | 2017-2020 | 74,433 | PIB (2017-2020) |
| News | PIB Archives | 2015-2016 | 27,383 | PIB Archives (2015-2016) |
| News | Prothomalo | 2014-2020 | 283,947 | Prothomalo (2014-2020) |
| News | DW News | 2004-2020 | 23,054 | DW News (2004-2020) |
| News | OneIndia | Until 2021-03 | 34,453 | OneIndia (20210320) |
| Sports | MyKhel | Until 2021-03 | 23,971 | MyKhel (20210320) |
| Sports | kolkata24x7 | Until 2021-03 | 29,318 | kolkata24x7-Sports (20210320) |
| Technology | kolkata24x7 | Until 2021-03 | 17,083 | kolkata24x7-Technology (20210320) |
| News | kolkata24x7 | Until 2021-03 | 49,633 | kolkata24x7-News (20210320) |
| General | kolkata24x7 | Until 2021-03 | 3,215 | kolkata24x7-General (20210320) |
| Entertainment | kolkata24x7 | Until 2021-03 | 19,134 | kolkata24x7-Entertainment (20210320) |
| Education | kolkata24x7 | Until 2021-03 | 2,035 | kolkata24x7-Education (20210320) |
| Financial | kolkata24x7 | Until 2021-03 | 2,640 | kolkata24x7-Financial (20210320) |
| Lifestyle | kolkata24x7 | Until 2021-03 | 1,517 | kolkata24x7-Lifestyle (20210320) |
| News | AsianetNews | Until 2021-03 | 18,423 | AsianetNews (20210320) |
| News | The Sentinel | Until 2021-03 | 6,800 | The Sentinel (20210320) |
| News | Times of India | Until 2021-03 | 71,969 | TOI-News (20210320) |
| News | Zee News | Until 2021-04 | 76,834 | Zee-News (2011-2021) |
| General | Multiple sources* | NA | 223,574 | General Corpus |
* - General corpus from Wikipedia, Custom NER, Wikinews, PMIndia etc.
| Domain | Source | Year | En-Ta pairs count | Corpus Download Link |
|---|---|---|---|---|
| Legal | Multiple Sources | 2010-2020 | 816,729 | Indian Judiciary |
| Legal | HC/SUVAS | NA | 10,851 | HC/SUVAS |
| Legal | Legal Terminologies | NA | 3,595 | Legal Terms |
| General | Nouns Dictionary | NA | 62,920 | Nouns Dict |
| News | PIB | 2017-2020 | 104,836 | PIB (2017-2020) |
| News | PIB Archives | 2014-2016 | 23,281 | PIB Archives (2014-2016) |
| General | Wikipedia | Until 2021-02 | 74,628 | Wikipedia (20210201) |
| Tourism | Native Planet | Until 2021-03 | 24,730 | Native Planet (20210320) |
| News | OneIndia | Until 2021-03 | 61,937 | OneIndia (20210320) |
| Sports | MyKhel | Until 2021-03 | 35,390 | MyKhel (20210320) |
| Automobile | DriveSpark | Until 2021-03 | 65,601 | DriveSpark (20210303) |
| Financial | GoodReturns | Until 2021-03 | 22,961 | GoodReturns-Financial (20210320) |
| Sports | Indian Express | Until 2021-03 | 9,994 | IE-Sports (20210320) |
| Technology | Indian Express | Until 2021-03 | 18,651 | IE-Technology (20210320) |
| News | Indian Express | Until 2021-03 | 26,235 | IE-News (20210320) |
| Lifestyle | Indian Express | Until 2021-03 | 4,356 | IE-Lifestyle (20210320) |
| Entertainment | Indian Express | Until 2021-03 | 8,557 | IE-Entertainment (20210320) |
| Education | Indian Express | Until 2021-03 | 3,277 | IE-Education 20210320) |
| Financial | Indian Express | Until 2021-03 | 832 | IE-Financial (20210320) |
| News | AsianetNews | Until 2021-03 | 24,553 | AsianetNews (20210320) |
| News | Times of India | Until 2021-03 | 39,835 | TOI-News (20210320) |
| News | Zee News | Until 2021-04 | 79,105 | Zee-News (2011-2021) |
| Domain | Source | Year | En-Ml pairs count | Corpus Download Link |
|---|---|---|---|---|
| Legal | Multiple Sources | 2010-2020 | 1,029,757 | Indian Judiciary |
| Legal | HC/SUVAS | NA | 8,222 | HC/SUVAS |
| Legal | Legal Terminologies | NA | 3,377 | Legal Terms |
| General | Nouns Dictionary | NA | 66,450 | Nouns Dict |
| News | PIB | 2017-2020 | 27,538 | PIB (2017-2020) |
| News | PIB Archives | 2014-2016 | 13,113 | PIB Archives (2014-2016) |
| General | Wikipedia | Until 2021-02 | 44,370 | Wikipedia (20210201) |
| News | OneIndia | Until 2021-03 | 48,891 | OneIndia (20210320) |
| Sports | MyKhel | Until 2021-03 | 26,526 | MyKhel (20210320) |
| Automobile | DriveSpark | Until 2021-03 | 49,663 | DriveSpark (20210303) |
| Tourism | Native Planet | Until 2021-03 | 27,024 | Native Planet (20210320) |
| Financial | GoodReturns | Until 2021-03 | 16,325 | GoodReturns(20210320) |
| Sports | Indian Express | Until 2021-03 | 13,305 | IE-Sports (20210320) |
| Technology | Indian Express | Until 2021-03 | 15,318 | IE-Technology (20210320) |
| News | Indian Express | Until 2021-03 | 36,337 | IE-News (20210320) |
| General | Indian Express | Until 2021-03 | 5,972 | IE-General (20210320) |
| Entertainment | Indian Express | Until 2021-03 | 12,430 | IE-Entertainment (20210320) |
| Financial | Indian Express | Until 2021-03 | 345 | IE-Financial (20210320) |
| Financial | MarketFeed | Until 2021-03 | 5,431 | MarketFeed (20210320) |
| News | AsianetNews | Until 2021-03 | 8,932 | AsianetNews (20210320) |
| News | Times of India | Until 2021-03 | 27,499 | TOI-News (20210320) |
| News | Zee News | Until 2021-04 | 18,935 | Zee-News (2011-2021) |
| Domain | Source | Year | En-Te pairs count | Corpus Download Link |
|---|---|---|---|---|
| Legal | Multiple Sources | 2010-2020 | 1,010,463 | Indian Judiciary |
| Legal | HC/SUVAS | NA | 5,712 | HC/SUVAS |
| Legal | Legal Terminologies | NA | 3,595 | Legal Terms |
| General | Nouns Dictionary | NA | 63,802 | Nouns Dict |
| News | PIB | 2017-2020 | 65,842 | PIB (2017-2020) |
| News | PIB Archives | 2014-2016 | 15,734 | PIB Archives (2015-2016) |
| General | Wikipedia | Until 2021-02 | 82,257 | Wikipedia (20210201) |
| News | OneIndia | Until 2021-03 | 71,125 | OneIndia (20210320) |
| Sports | MyKhel | Until 2021-03 | 20,912 | MyKhel (20210320) |
| Automobile | DriveSpark | Until 2021-03 | 67,688 | DriveSpark (20210303) |
| Tourism | Native Planet | Until 2021-03 | 40,733 | Native Planet (20210320) |
| Financial | GoodReturns | Until 2021-03 | 17,159 | GoodReturns (20210320) |
| News | Sakshi | Until 2021-03 | 113,458 | Sakshi (20210320) |
| News | AsianetNews | Until 2021-03 | 10,054 | AsianetNews (20210320) |
| News | Times of India | Until 2021-03 | 49,233 | TOI-News (20210320) |
| News | Zee News | Until 2021-04 | 23,207 | Zee-News (2011-2021) |
| Domain | Source | Year | En-Kn pairs count | Corpus Download Link |
|---|---|---|---|---|
| Legal | Multiple Sources | 2010-2020 | 859,455 | Indian Judiciary |
| Legal | Legal Terminologies | NA | 3,356 | Legal Terms |
| General | Nouns Dictionary | NA | 54,126 | Nouns Dict |
| News | PIB | 2017-2020 | 50,731 | PIB (2017-2020) |
| News | PIB Archives | 2015-2016 | 20,910 | PIB Archives (2015-2016) |
| News | OneIndia | Until 2021-03 | 49,622 | OneIndia (20210320) |
| Sports | MyKhel | Until 2021-03 | 30,416 | MyKhel (20210320) |
| Automobile | DriveSpark | Until 2021-03 | 57,151 | DriveSpark (20210303) |
| Tourism | Native Planet | Until 2021-03 | 31,654 | Native Planet (20210320) |
| Financial | GoodReturns | Until 2021-03 | 15,302 | GoodReturns (20210320) |
| News | AsianetNews | Until 2021-03 | 37,204 | AsianetNews (20210320) |
| News | Times of India | Until 2021-03 | 76,763 | TOI-News (20210320) |
| News | Zee News | Until 2021-04 | 65,327 | Zee-News (2011-2021) |
| Domain | Source | Year | En-Mr pairs count | Corpus Download Link |
|---|---|---|---|---|
| Legal | Multiple Sources | 2010-2020 | 1,205,571 | Indian Judiciary |
| Legal | Legal Terminologies | NA | 1,052 | Legal Terms |
| General | Nouns Dictionary | NA | 57,307 | Nouns Dict |
| News | PIB | 2017-2020 | 155,647 | PIB (2017-2020) |
| News | PIB Archives | 2015-2016 | 28,622 | PIB Archives (2015-2016) |
| Sports | Indian Express | Until 2021-03 | 51,857 | IE-Sports (20210320) |
| News | Indian Express | Until 2021-03 | 110,823 | IE-News (20210320) |
| Lifestyle | Indian Express | Until 2021-03 | 2,346 | IE-Lifestyle (20210320) |
| Entertainment | Indian Express | Until 2021-03 | 49,821 | IE-Entertainment (20210320) |
| Financial | Indian Express | Until 2021-03 | 6,869 | IE-Financial (20210320) |
| News | Times of India | Until 2021-03 | 581,693 | TOI-News (2008-2020) |
| News | The Wire | Until 2021-03 | 25,084 | The Wire (20210320) |
| Entertainment | Lokmat | Until 2021-05 | 8,738 | Lokmat-Entertainment (20210501) |
| News | Lokmat | Until 2021-05 | 77,811 | Lokmat-News (20210501) |
| Lifestyle | Lokmat | Until 2021-05 | 2,686 | Lokmat-Lifestyle (20210501) |
| Sports | Lokmat | Until 2021-05 | 33,025 | Lokmat-Sports (20210501) |
| Technology | Lokmat | Until 2021-05 | 2,943 | Lokmat-Technology (20210501) |
| Financial | Lokmat | Until 2021-05 | 5,509 | Lokmat-Financial (20210501) |
| Healthcare | Lokmat | Until 2021-05 | 4,488 | Lokmat-Healthcare (20210501) |
| News | Zee News | Until 2021-04 | 125,961 | Zee-News (2011-2021) |
| Domain | Source | Year | En-Pa pairs count | Corpus Download Link |
|---|---|---|---|---|
| Legal | Multiple Sources | 2010-2020 | 1,261,948 | Indian Judiciary |
| Legal | HC/SUVAS | NA | 10,978 | HC/SUVAS |
| Legal | Legal Terminologies | NA | 3,578 | Legal Terms |
| General | Nouns Dictionary | NA | 53,665 | Nouns Dict |
| News | PIB | 2017-2020 | 204,575 | PIB (2017-2020) |
| News | PIB Archives | 2015-2016 | 30,923 | PIB Archives (2015-2016) |
| News | Tribune | Until 2021-03 | 485,333 | Tribune (20210320) |
| Government Press Release | Punjab Govt | As of 2021-03 | 32,589 | Punjab Govt (20210320) |
| News | Jagran | Until 2021-03 | 34,162 | Jagran-News (20210320) |
| Technology | Jagran | Until 2021-03 | 6,524 | Jagran-Technology (20210320) |
| Education | Jagran | Until 2021-03 | 3,155 | Jagran-Education (20210320) |
| Entertainment | Jagran | Until 2021-03 | 7,565 | Jagran-Entertainment (20210320) |
| Financial | Jagran | Until 2021-03 | 7,741 | Jagran-Financial (20210320) |
| Sports | Jagran | Until 2021-03 | 9,034 | Jagran-Sports (20210320) |
| Lifestyle | Jagran | Until 2021-03 | 2,528 | Jagran-Lifestyle (20210320) |
| Domain | Source | Year | En-Gu pairs count | Corpus Download Link |
|---|---|---|---|---|
| Legal | Multiple Sources | 2010-2020 | 1,227,233 | Indian Judiciary |
| Legal | Legal Terminologies | NA | 3,575 | Legal Terms |
| General | Nouns Dictionary | NA | 55,037 | Nouns Dict |
| News | PIB | 2017-2020 | 104,591 | PIB (2017-2020) |
| News | PIB Archives | 2015-2016 | 29,443 | PIB Archives (2015-2016) |
| News | OneIndia | Until 2021-03 | 48,325 | OneIndia (20210320) |
| News | Times of India | Until 2021-03 | 85,777 | TOI-News (2008-2020) |
| Government Press Release | Gujarat Govt | As of 2021-03 | 5,260 | Gujarat Govt (20210320) |
| News | Zee News | Until 2021-04 | 97,375 | Zee-News (2011-2021) |
| Domain | Source | Year | En-As pairs count | Corpus Download Link |
|---|---|---|---|---|
| News | PIB | 2017-2020 | 38,100 | PIB (2017-2020) |
| News | PIB Archives | 2015-2016 | 6,188 | PIB Archives (2015-2016) |
| News | The Sentinel | Until 2021-03 | 2,886 | The Sentinel (20210320) |
| Domain | Source | Year | En-Ur pairs count | Corpus Download Link |
|---|---|---|---|---|
| News | PIB | 2017-2020 | 141,246 | PIB (2017-2020) |
| News | PIB Archives | 2015-2016 | 9,939 | PIB Archives (2015-2016) |
| News | The Wire | Until 2021-03 | 10,863 | The Wire (20210320) |
| Domain | Source | Year | En-Or pairs count | Corpus Download Link |
|---|---|---|---|---|
| News | PIB | 2017-2020 | 73,020 | PIB (2017-2020) |
| News | PIB Archives | 2015-2016 | 19,289 | PIB Archives (2015-2016) |