feat: add ftchinese parser #264
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I've been looking into character encoding. I think cheerio encodes all entities when calling In order to not encode all entities, the return normalizeSpaces($.html(node));To this: return normalizeSpaces($.html(node, { decodeEntities: false }));In my use of mercury, I sanitize the output from mercury so this would be fine, but I'm guessing mercury wants to not pass on executable javascript? I saw that another project switched to JSDOM to avoid this. Not sure if that would be a suitable replacement for mercury. /cc @adampash |
|
@benubois thanks for the input, this reminds that I did use this method in my other project and it works flawlessly. Unfortunately, JSDOM is less performant than cheerio from my past experience. |
Actually, I think this is probably fine. Mercury sanitizes tags and attributes, and it looks like you're right, this could fix a lot o these problems. If someone wants to submit a PR with a test that adds the I should also follow up to say that if you do happen to see any significant security vulnerabilities in doing so, I'm all ears. |
adampash
reviewed
Feb 13, 2019
|
@adampash adding https://github.com/postlight/mercury-parser/blob/e033835c7287904371371f922c487e6d0d7d7db8/src/resource/index.js#L63-L79 will skip the decoding of other than |
I think we want to keep that |
|
I did change that but it has no effect on Chinese characters. |
|
@adampash I've come to believe that Here's my pull request addressing this. |
|
There is another solution for Chinese, see https://github.com/HenryQW/OpenCC.henry.wang/blob/master/route.js#L7-L21 Not sure how would this merge into mercury-parser though, probably as a middleware? |
🤖 Automated Parsing Preview 🤖Commit: Original Article | HTML Fixture | Parsed Content Preview Parsed JSON{
"title": "英国认为华为风险可控",
"content": "<div><div id=\"story-body-container\"> <p>英国政府得出结论认为，它能够缓解在5G网络中使用华为(Huawei)设备的风险。这个结论沉重打击了美国说服盟国把华为挡在高速电信系统门外的努力。</p><p>两位知情人士将这一尚未公开的结论告诉英国《金融时报》，称英国国家网络安全中心(National Cyber Security Centre, NCSC)认定，有办法限制在未来5G超高速网络中使用华为设备的风险。</p><p>这一结论出炉之际，美国正加紧努力说服盟国禁止华为参与电信网络建设，理由是这家中国供应商可能帮助中国政府从事间谍活动或网络破坏。</p><p>美国国家安全局(NSA)近来与盟友和合作伙伴分享更多信息以强调相关风险，但是数个欧洲国家（包括英国和德国）并未被说服需要实施禁令。</p><p>一位熟悉相关争论的人士表示，英国的结论对欧洲各国领导人将“有很大分量”，因为英国是五眼联盟(Five Eyes)情报分享网络的成员，可以获得敏感的美国情报。</p><p>“其他国家可以提出这样的论点，即如果英国人有信心缓解国家安全威胁，那么他们也可以向国内公众和美国行政当局保证，只要他们采取了英国人推荐的各种预防措施，他们继续允许国内电信服务提供商使用中国组件就是审慎的，”此人表示。</p><p>美国提出，5G的速度将如此之快——并且有如此多的军事用途——以至于使用任何中国电信设备都带有太高的风险。美国官员还提出，虽然到目前为止可能没有恶意活动的证据，但华为可能会使用恶意软件更新为间谍活动创造条件。</p><p>英国信号情报机构——政府通信总部(GCHQ)前主任罗伯特•汉尼根(Robert Hannigan)最近在英国《金融时报》撰文，称NCSC“从未发现任何证据证明中国政府通过华为进行任何恶意网络活动”，而且“有关在5G网络的任何部分采用任何中国技术都代表着不可接受的风险的断言是无稽之谈”。</p><p>英国的结论与同为五眼成员的澳大利亚和新西兰形成鲜明对比，后两国去年就已禁止本国电信提供商在5G网络中使用华为设备。</p><p>与此同时，唐纳德•特朗普(Donald Trump)正在考虑发布实际上将禁止美国公司使用华为设备的行政命令。熟悉这道命令的一名人士表示，它将以“不对公司和国家点名”的方式写成。</p><p>美国副总统迈克•彭斯(Mike Pence)上周六在慕尼黑安全会议(Munich Security Conference)上发表演讲时指出，由于中国法律要求电信公司与中国政府共享数据，因此华为构成威胁。</p><p>在同一个论坛上，北约(NATO)秘书长延斯•斯托尔滕贝格(Jens Stoltenberg)告诉英国《金融时报》，北约联盟“非常认真地”对待围绕华为的担忧，数个盟国希望拿出协调一致的回应。</p><p>“我们必须看一看我们需要达到的回应协调水平。我们作为一个联盟还没有得出结论，但这表明了需要应对这个问题，”他表示。</p><p>英国秘密情报局（SIS，通称“军情六处”，即MI6）局长亚历克斯•扬格(Alex Younger)上周五表示，英国可能会对华为采取比美国更为温和的态度，称这个问题过于复杂，不宜简单地封杀该公司。他表示，这是“一个比‘进来还是出去’更加复杂的问题”，而且各国拥有“找到所有这些问题的答案的主权权利”。</p><p>对于已经认定使用华为设备的风险可控的说法，NCSC没有表示异议。</p></div></div>",
"author": null,
"date_published": "2018-01-02T07:17:00.000Z",
"lead_image_url": "http://i.ftimg.net/picture/3/000082493_piclink.jpg",
"dek": "英国政府得出结论认为，能够缓解在5G网络中使用华为设备的风险。这个尚未公开的结论沉重打击美国的游说努力。",
"next_page_url": null,
"url": "http://www.ftchinese.com/story/001081496?full=y",
"domain": "www.ftchinese.com",
"word_count": 13,
"direction": "ltr",
"total_pages": 1,
"rendered_pages": 1
}
3 failed tests 😱WwwFtchineseComExtractor initial test case returns the author See what went wrong AssertionError [ERR_ASSERTION]: null == '英国《金融时报》 迪米 华盛顿 ， 戴维•邦德 慕尼黑报道'
at Object.equal (/home/circleci/project/src/extractors/custom/www.ftchinese.com/index.test.js:48:14)
at tryCatch (/home/circleci/project/node_modules/regenerator-runtime/runtime.js:62:40)
at Generator.invoke [as _invoke] (/home/circleci/project/node_modules/regenerator-runtime/runtime.js:288:22)
at Generator.prototype.(anonymous function) [as next] (/home/circleci/project/node_modules/regenerator-runtime/runtime.js:114:21)
at asyncGeneratorStep (/home/circleci/project/src/extractors/custom/www.ftchinese.com/index.test.js:17:103)
at _next (/home/circleci/project/src/extractors/custom/www.ftchinese.com/index.test.js:19:194)
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:188:7)WwwFtchineseComExtractor initial test case returns the date_published See what went wrong AssertionError [ERR_ASSERTION]: null == '2018-01-02T07:17:00.000Z'
at Object.equal (/home/circleci/project/src/extractors/custom/www.ftchinese.com/index.test.js:61:14)
at tryCatch (/home/circleci/project/node_modules/regenerator-runtime/runtime.js:62:40)
at Generator.invoke [as _invoke] (/home/circleci/project/node_modules/regenerator-runtime/runtime.js:288:22)
at Generator.prototype.(anonymous function) [as next] (/home/circleci/project/node_modules/regenerator-runtime/runtime.js:114:21)
at asyncGeneratorStep (/home/circleci/project/src/extractors/custom/www.ftchinese.com/index.test.js:17:103)
at _next (/home/circleci/project/src/extractors/custom/www.ftchinese.com/index.test.js:19:194)
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:188:7)WwwFtchineseComExtractor initial test case returns the content See what went wrong AssertionError [ERR_ASSERTION]: '' == '英国政府得出结论认为，它能够缓解在5G网络中使用华为(Huawei)设备的风险。这个结论沉重打击了美国说服盟国把华为挡在高速电信系统门外的努力。两位知情人士将这一尚未公开的结论告诉英国《金融时报》，称英国国家网络安全中心(National'
at Object.equal (/home/circleci/project/src/extractors/custom/www.ftchinese.com/index.test.js:105:14)
at tryCatch (/home/circleci/project/node_modules/regenerator-runtime/runtime.js:62:40)
at Generator.invoke [as _invoke] (/home/circleci/project/node_modules/regenerator-runtime/runtime.js:288:22)
at Generator.prototype.(anonymous function) [as next] (/home/circleci/project/node_modules/regenerator-runtime/runtime.js:114:21)
at asyncGeneratorStep (/home/circleci/project/src/extractors/custom/www.ftchinese.com/index.test.js:17:103)
at _next (/home/circleci/project/src/extractors/custom/www.ftchinese.com/index.test.js:19:194)
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:188:7) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It seems the parser can't handle Chinese characters for content, but works for title, author and dek.
Preview is showing a bunch of unicode rather than the actual Chinese characters.
{ "title": "欧盟否决西门子-阿尔斯通铁路业务合并案", "content": "<div><div id=\"story-body-container\"><p>德国和法国铁路设备制造商西门子(Siemens)与阿尔斯通(Alstom)之间的合并已被欧盟竞争执法机构否决，监管者不接受这样一个观点，即这一联姻对于欧洲与由国家支持的中国竞争对手竞争是必要的。</p><p>欧盟竞争事务专员玛格丽特•维斯特格(Margrethe Vestager)周三宣布了这项否决。此前，其他欧盟专员以及各国监管机构支持她的立场，即这笔交易可能导致价格上涨并限制选择，最终可能有损欧洲乘客的利益。</p><p>“如果没有足够的补救措施，这一合并将导致保证乘客安全以及用于下一代高速列车的信号系统价格上涨，”维斯特格表示。她补充称：“两家公司不愿意应对我们严重的竞争担忧。”</p><p>这一否决将加剧一场长期辩论，焦点问题是欧盟严格的反垄断规则是否需要更新，以便让欧盟企业与由国家支持的中国企业展开竞争。在各自政府的支持下，西门子和阿尔斯通表示，合并其铁路业务对于抵御来自中国中车(CRRC)的竞争是必要的；后者是世界最大的铁路车辆制造商。</p><p>然而，竞争官员们得出结论认为，中国企业不太可能很快在欧洲销售火车，这意味着欧盟严格的反垄断规则要求这两家公司出售可观的火车和信号资产。</p><p>法国财政部长布鲁诺•勒梅尔(Bruno Le Maire)形容维斯特格的决定“是一个政治错误：欧盟委员会的职责是捍卫欧洲的经济利益……拒绝阿尔斯通与西门子合并将有利于中国的经济和产业利益”。这些情绪得到德国经济部长彼得•阿尔特迈尔(Peter Altmaier)的呼应，他呼吁修改欧盟竞争规则，使大型并购交易变得更加容易。</p><p>西门子周三表示了失望。“保护本地顾客利益绝不能意味着欧洲无法与中国、美国等领先国家同处一个公平的竞争环境，”该公司首席执行官凯飒(Joe Kaeser)表示。</p><p>阿尔斯通表示，这对于欧洲工业是一个“明显的挫折”。阿尔斯通首席执行官亨利•波帕-拉法基(Henri Poupart-Lafarge)表示：“如果说我有一个遗憾，那就是我用了‘欧洲冠军企业’这个短语。”</p><p>然而，英国、西班牙、荷兰和比利时的国家监管机构一致公开表示这两家公司提议的补救措施远远不够，并对这一否决表示欢迎。负责监管英国铁路的英国铁路监管办公室(Office of Rail Regulation)表示：“我们从一开始就明确表示，这次（合并）对英国乘客、货运客户和纳税人来说都是一笔糟糕的交易。”</p><p>智库布鲁盖尔(Bruegel)的访问研究员马修•海姆(Mathew Heim)说：“支持欧洲冠军企业不是欧洲竞争法律的职能。</p><p>“相反，欧洲需要一项连贯的计划，以培育在全球竞争环境下能够蓬勃发展、而无需诉诸保护主义的关键欧洲产业。”</p><p>西门子-阿尔斯通提议合并之际，欧洲正处于一个政治过渡期。今年5月的欧盟选举，以及今年布鲁塞尔领导层的变动，可能带来改革竞争政策的机会。</p><p>欧洲反垄断主管承认中国国有企业构成经济威胁，同时为竞争规则做了辩护——她辩称，这些规则帮助欧盟公司保持创新能力和国际竞争力。</p><p>维斯特格指出，欧盟有其他手段来制定产业政策，如外来投资筛查、反补贴与贸易防御措施，以及推动封闭的市场提供互惠准入。</p><p>阿尔斯通的波帕-拉法基周二告诉《费加罗报》(Le Figaro)，不会第二次尝试合并，两家公司今后将各走各的路。</p><p><i>帕特里克•麦吉(Patrick McGee)法兰克福、戴维•基奥恩(David Keohane)巴黎补充报道</i></p><p>译者/和风</p></div></div>", "author": "罗谢尔•托普兰斯基", "date_published": "2019-01-02T06:21:00.000Z", "lead_image_url": "http://i.ftimg.net/picture/7/000083507_piclink.jpg", "dek": "欧盟反垄断专员没有接受西门子和阿尔斯通提出的合并理由，即这次合并对于欧洲企业与中国国有竞争对手展开竞争是必要的。", "next_page_url": null, "url": null, "domain": null, "excerpt": null, "word_count": null, "direction": null, "total_pages": 1, "rendered_pages": 1 }Can be reproduced via using qdaily extractor too.