Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check encode type for doanwload task #269

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Check encode type for doanwload task #269

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
eaa1ff3
check encode type for download task
DisaPadla Oct 20, 2024
6571deb
add unit test for detect encode type
DisaPadla Oct 20, 2024
6bbc6d6
use regex for detecting encode type
DisaPadla Oct 20, 2024
d8fd553
run formatter
DisaPadla Oct 20, 2024
e078af4
run formatter
DisaPadla Oct 20, 2024
a236ddd
rename detectEncodeType to parseEncodeType
DisaPadla Oct 21, 2024
d85ecb8
improve regex for parseEncodeType
DisaPadla Oct 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion core/download.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,12 @@ function detectType(text: string): string | undefined {
}
}

function parseEncodeType(response: Partial<Response>): string {
let headers = response.headers ?? new Headers()
let contentType = headers.get('content-type')?.toLowerCase() ?? ''
return contentType.match(/;\s*charset=([^\s;]+)/)?.[1] ?? 'utf-8'
}

export function createTextResponse(
text: string,
other: Partial<Omit<TextResponse, 'ok' | 'text'>> = {}
Expand Down Expand Up @@ -122,7 +128,16 @@ export function createDownloadTask(): DownloadTask {
},
async text(url, opts) {
let response = await this.request(url, opts)
let text = await response.text()
let text: string
let encodeType = parseEncodeType(response)
if (encodeType !== 'utf-8') {
let buffer = await response.arrayBuffer()
let decoder = new TextDecoder(encodeType)
text = decoder.decode(buffer)
} else {
text = await response.text()
}

if (controller.signal.aborted) {
throw new DOMException('', 'AbortError')
}
Expand Down
16 changes: 16 additions & 0 deletions core/test/download.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,22 @@ test('can download text by keeping eyes on abort signal', async () => {
await rejects(response2, (e: Error) => e.name === 'AbortError')
})

test('detect unpopular encode type', async () => {
let task = createDownloadTask()

expectRequest('https://example.com').andRespond(
200,
'Hi',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test will pass even if we haven’t encoding feature.

What do you think if we will use CP1251 and put that broken (when CP1251 was parsed as UTF-8) encoding text here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it is too long review, I can do it myself

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i will check later

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm, if i just cope paste broken text (like �), it doesnt work correctly. Continue research

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Try binary format for symbol

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another way is to get JS code for UTF-8→CP1251 convertation and put:

to1251('тест')

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, what do you mean "Try binary format for symbol"?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, what do you mean "Try binary format for symbol"?

String.fromCodePoint

'text/plain; charset=shift-jis'
)
let response1 = await task.text('https://example.com')

equal(response1.ok, true)
equal(response1.status, 200)
equal(response1.url, 'https://example.com')
equal(response1.text, 'Hi')
})

test('parses XML content', async () => {
let text = createTextResponse('<html><body>Test</body></html>')
equal(text.parseXml()!.firstChild?.lastChild?.nodeName, 'BODY')
Expand Down