Ideosphere Forum

fx-discuss: Re: fx-discuss: Fwd: FX Claim: Tran – Machine tran slation by 2015

Author: Neal Gafter
Conversation: fx-discuss: Fwd: FX Claim: Tran – Machine translation by 2015 ( prev | next ) reply!
Topic: fx-discuss ( prev | next )
In-Reply-To:
Followed-Up-By: Neal Gafter's post
Date: Sat Jan 20, 2018 04:39 pm
chrisran.bma e-mail
Neal Gafter
Neal Gafter



I am still around. I will start looking at this in a few days.

On Sat, Jan 20, 2018 at 9:18 AM chrisran.bma e-mail <chrisran.bma@virgin.net>
wrote:

> Hi,
>
> The due date for this claim 2017/12/31 has passed.
>
> Does anyone think or preferably have evidence to suggest this claim is
> true or false?
>
> Is the judge, Neal Gafter aka loophole still around?
>
>
>
> I have previously written things like:
>
>
> Findings of the 2017 Conference on Machine Translation (WMT17)
>
> http://www.statmt.org/wmt17/pdf/WMT17.pdf
>
> (September 7-8, 2017)
>
> and
>
> Findings of the 2016 Conference on Machine Translation (WMT16)
>
> http://www.aclweb.org/anthology/W16-2301
>
> (11-12 August 2016)
>
> Perhaps the following looks like what we want:
>
> [quote]
>
> 5.5.2 Human evaluation results
>
> Table 35 includes DA results for English-German and Table 36 shows results
> for German-English APE systems. Clusters are identified by grouping systems
> together according to which systems significantly outperform all others in
> lower ranking clusters, according to Wilcoxon rank-sum test.
>
> # Ave % Ave z System
>
> ___________________________
>
> − 84.8 0.520 HUMAN POST EDIT
>
> ___________________________
>
> 1 78.2 0.261 AMU
>
> 77.9 0.261 FBK
>
> 76.8 0.221 DCU
>
> ___________________________
>
> 4 73.8 0.115 JXNU
>
> ___________________________
>
> 5 71.9 0.038 USAAR
>
> 71.1 0.014 CUNI
>
> 70.2 −0.020 LIG
>
> ___________________________
>
> − 68.6 −0.083 NO POST EDIT
>
> Table 35: EN-DE DA Human evaluation results showing average raw DA scores
> (Ave %) and average standardized scores (Ave z), lines between systems
> indicate clusters according to Wilcoxon rank-sum test at p-level p ≤ 0.05.
>
> [/quote]
>
> Seems to indicate that human translation is better than machine
> translation, but of course that doesn't guarantee that there isn't a better
> translation program somewhere from pre 31 Dec 2015 that simply didn't
> attend the conference.
>
> Still if human level translation existed in 2015, you would not expect to
> read things like
>
> [quote]
>
> This steady improvement has been mainly driven by the massive migration to
> the neural approach, which in 2016 allowed the winning system to achieve
> impressive results
>
> [/quote]
>
> I don't believe there is a program that can justifiably claim "equal or
> better average quality, as professional human translations" but proving a
> negative is difficult. I suggest if there was such a program it would be
> big news, not difficult to find, and conference findings would be markedly
> different to those linked above.
>
> Not sure how much more a judge might want before deciding how to judge the
> claim. Are there any more authoritative events or other event before claim
> deadline of 31 Dec 2017? (Note program has to exist by 31 Dec 2015 and
> translations have to 'be of comparable cost and turnaround time'.)
>
>
> The comparable cost and turnaround time requirement seems to me to
> indicate that secret research would not qualify.
>
>
> Seems like an obvious false to me.
>
> Regards
>
> Chris Randles
>
> (crandles 7886)
>
> Disclosure I hold -3603 in this claim
>
>
>
>

source



All trademarks, copyrights, and messages on this page are owned by their respective owners.
Forum: Copyright (c) 2000-2001 Javien Inc All rights reserved. Distributed under the GPL