A Web-based English Proofing System for English as a Second Language Users

Users

XingYi1,JianfengGao2andWilliamB.Dolan2

1

CenterforIntelligentInformationRetrieval,DepartmentofComputerScience

UniversityofMassachusetts,Amherst,MA01003-4610,USA

yixing@cs.umass.edu

2

MicrosoftResearch,OneMicrosoftWay,Redmond,WA98052,USA

{jfgao,billdol}@microsoft.com

Abstract

Wedescribeanalgorithmthatreliesonwebfrequencycountstoidentifyandcorrectwritingerrorsmadebynon-nativewritersofEnglish.Evaluationofthesystemonareal-worldESLcorpusshowedverypromisingperformanceontheverydifficultproblemofcritiquingEnglishdetermineruse:62%pre-cisionand41%recall,withafalseflagrateofonly2%(comparedtoarandom-guessingbaselineof5%precision,7%recall,andmorethan80%falseflagrate).Performanceoncollocationerrorswaslessgood,sug-gestingthataweb-basedapproachshouldbecombinedwithlocallinguisticresourcestoachievebotheffectivenessandefficiency.

SearchPhraseEnglishasSecondLanguage

EnglishasaSecondLanguage

Google.com306,0001,490,000

Live.com52,40738,336,308

Yahoo.com386,0004,250,000

Table1:WebHitsforPhrasalUsagesrawandeditedESLproseposeanobstacletothisapproach.

InthisworkweconsidertheprospectofusingtheWeb,withitsbillionsofwebpages,asadatasourcewiththepotentialtoaidESLwriters.OurresearchismotivatedbytheobservationthatESLusersalreadyusetheWebasacorpusofgoodEn-glish,oftenusingsearchenginestodecidewhetheraparticularspelling,phrase,orsyntacticconstruc-tionisconsistentwithusagefoundontheWeb.Forexample,unsurewhetherthenative-soundingphraseincludesthedeterminer“a”,ausermightsearchforbothquotedstrings“EnglishasSecondLanguage”and“EnglishasaSecondLanguage”.ThecountsobtainedforeachofthesephrasesonthreedifferentsearchenginesareshowninTable1.Notethecor-rectversion,“EnglishasaSecondLanguage”,hasamuchhighernumberofwebhits.

Inordertodeterminewhetherthisapproachholdspromise,weimplementedaweb-basedsystemforESLwritingerrorproofing.Thispilotstudywasin-tendedto:

1.identifydifferenttypesofESLwritingerrorsandhowoftentheyoccurinESLusers’writingsamples,sothatthechallengesanddifficultiesofESLerrorproofingcanbeunderstoodbetter;

2.exploretheadvantagesanddrawbacksofaweb-

1Introduction

ProofingtechnologyfornativespeakersofEnglishhasbeenafocusofworkfordecades,andsometoolslikespellcheckersandgrammarcheckershavebecomestandardfeaturesofdocumentprocessingsoftwareproducts.However,designinganEnglishproofingsystemforEnglishasaSecondLanguage(ESL)userspresentsamajorchallenge:ESLwrit-ingerrorsvarygreatlyamonguserswithdifferentlanguagebackgroundsandproficiencylevels.Re-centworkbyBrockettetal.(2006)utilizedphrasalStatisticalMachineTranslation(SMT)techniquestocorrectESLwritingerrorsanddemonstratedthatthisdata-intensiveSMTapproachisverypromising,buttheyalsopointedoutSMTapproachreliesontheavailabilityoflargeamountoftrainingdata.Theex-penseanddifficultyofcollectinglargequantitiesof

basedapproach,discoverusefulwebdatafeatures,andidentifywhichtypesofESLerrorscanbereli-ablyproofedusingthistechnique.

WefirstcatalogsomemajorcategoriesofESLwritingerrors,thenreviewrelatedwork.Section3describesourWeb-basedEnglishProofingSystemforESLusers(calledESL-WEPSlater).Section4presentsexperimentalresults.Section5concludes.1.1ESLWritingErrors

InordertogetESLwritingsamples,weemployedathirdpartytoidentifylargevolumesofESLwebpages(mostlyfromJapanese,KoreanandChineseESLusers’blogs),andcull1Knon-nativesen-tences.AnativespeakerthenrewrotetheseESLsentences–whenpossible–toproduceanative-soundingversion.353(34.9%)oftheoriginal1012ESLsentenceswerelabeled“native-like”,another347(34.3%)wererewritten,andtheremaining312(30.8%)wereclassifiedassimplyunintelligible.Table2showssomeexamplesfromthecorpusil-lustratingsometypicaltypesofESLwritingerrorsinvolving:(1)Verb-NounCollocations(VNC)and(4)Adjective-NounCollocations(ANC);(2)incor-rectuseofthetransitiveverb“attend”;(3)deter-miner(article)usageproblems;and(5)morecom-plexlexicalandstyleproblems.Weanalyzedallthepre-andpost-editedESLsamplesandfound441ESLerrors:about20%aredeterminerusageprob-lems(missing/extra/misused);15%areVNCerrors,1%areANCerrors;othersrepresentcomplexsyn-tactic,lexicalorstyleproblems.Multipleerrorscanco-occurinonesentence.Theseshowthatreal-worldESLerrorproofingisverychallenging.

OurfindingsareconsistentwithpreviousresearchresultsonESLwritingerrorsintworespects:1.ESLusershavesignificantlymoreproblemswithdeterminerusagethannativespeakersbe-causetheuseandomissionofdefiniteandindefinitearticlesvariesacrossdifferentlan-guages(SchneiderandMcCoy,1998)(Lons-daleandStrong-Krause,2003).2.CollocationerrorsarecommonamongESLusers,andcollocationalknowledgecontributestothedifferencebetweennativespeakersandESLlearners(SheiandPain,2000):inCLEC,areal-worldChineseEnglishLearnerCorpus

(GuiandYang,2003),about30%ofESLwrit-ingerrorsinvolvedifferenttypesofcollocationerrors.

Intheremainderofthepaper,wefocusonproofingdeterminerusageandVNCerrors.

2RelatedWork

Researchershaverecentlyproposedsomesuccess-fullearning-basedapproachesforthedeterminerse-lectiontask(Minnenetal.,2000),butmostofthisworkhasaimedonlyathelpingnativeEnglishuserscorrecttypographicalerrors.Gamonetal.(2008)recentlyaddressedthechallengingtaskofproofingwritingerrorsforESLusers:theyproposecombin-ingcontextualspellertechniquesandlanguagemod-elingforproofingseveraltypesofESLerrors,anddemonstratesomepromisingresults.Inadeparturefromthiswork,oursystemdirectlyuseswebdatafortheESLerrorproofingtask.

ThereisasmallbodyofpreviousworkontheuseofonlinesystemsaimedathelpingESLlearnerscorrectcollocationerrors.InSheiandPain’ssys-tem(2000),forinstance,theBritishNationalCor-pus(BNC)isusedtoextractEnglishcollocations,andanESLlearnerwritingcorpusisthenusedtobuildacollocationErrorLibrary.InJianetal.’ssys-tem(2004),theBNCisalsousedasasourceofcol-locations,withcollocationinstancesandtranslationcounterpartsfromthebilingualcorpusidentifiedandshowntoESLusers.Incontrasttothisearlierwork,oursystemusesthewebasacorpus,withstringfre-quencycountsfromasearchengineindexusedtoin-dicatewhetheraparticularcollocationisbeingusedcorrectly.

3Web-basedEnglishProofingSystemfor

ESLUsers(ESL-WEPS)

ThearchitectureofESL-WEPS,whichconsistsoffourmaincomponents,isshowninFig.1.

ParseESLSentenceandIdentifyCheckPointsESL-WEPSfirsttagsandchunks(SangandBuck-holz,2000)theinputESLsentence1,andidenti-fiestheelementsofthestructuresinthesentencetobecheckedaccordingtocertainheuristics:when

1

Onein-houseHMMchunkertrainedonEnglishPennTree-bankwasused.

ID12345Pre-editingversionWhichteamcantakethechampion?IattendtoPyoungTaekUniversity.I’maJapaneseandstudyingInfoandComputerScienceatKeioUniversity.Herworksarekindaeroticbuttheywillneverarouseanyobscene,devilthoughtswhichmight

destroythesoulofthedesigner.

Ithinkitissobeautifultogothewayoftheologyandveryattractivetoo,especiallyintheareaofChristianity.Post-editingversionWhichteamwillwinthechampionship?IattendPyoungTaekUniversity.I’mJapaneseandstudyingInfoComputerScienceatKeioUniversity.Herworksarekindoferotic,buttheywillneverarouseanyobscene,evilthoughtswhichmight

destroythesoulofthedesigner.

Ithinkitissobeautifultogetintotheology,especiallyChristianity,whichattractsme.

Table2:Somepre-andpost-editingESLwritingsamples,BoldItaliccharactersshowwheretheESLerrorsareandhowtheyarecorrected/rewrittenbynativeEnglishspeaker.

ESL󰀀Sentences󰀀I am 󰀀learning economics󰀀at university.󰀀Pre-processing󰀀(POS Tagger and Chunk Parser)󰀀Identify󰀀Check Point󰀀[VP am/VBP 󰀀learning󰀀/󰀀VBG󰀀economics󰀀/NNS]󰀀Generate a set of queries, in order to󰀀search correct English usages from Web󰀀Queries:󰀀1. [economics at university] AND [learning]󰀀2. [economics] AND [at university] AND󰀀[learning]󰀀3. [economics] AND [university] AND󰀀[learning]󰀀[NP I/PRP] [VP am/VBP 󰀀learning󰀀/󰀀VBG󰀀 economics󰀀/NNS] [PP at/IN] [NP󰀀university/NN] ./.󰀀Use Web statistics to identify plausible errors, Collect Summaries, Mine collocations or󰀀determiner usages, Generate good suggestions and provide Web example sentences󰀀N-best suggestions:󰀀1. studying 194󰀀2. doing 12󰀀3. visiting 11󰀀

Search󰀀Engine󰀀Web Examples:󰀀Why Study Economics? - For Lecturers󰀀The design of open days, conferences and other events for school󰀀students 󰀀studying economics󰀀 and/or thinking of 󰀀studying economics at󰀀university󰀀. These could be held in a university, in a conference 󰀀󰀀http://whystudyeconomics.ac.uk/lecturers/󰀀Figure1:SystemArchitecture

checkingVNCerrors,thesystemsearchesforastructureoftheform(VP)(NP)or(VP)(PP)(NP)inthechunkedsentence;whencheckingdeterminerusage,thesystemsearchesfor(NP).Table3showssomeexamples.Forefficiencyandeffectiveness,theusercanspecifythatonlyonespecificerrortypebecritiqued;otherwiseitwillcheckbotherrortypes:firstdeterminerusage,thencollocations.

GenerateQueriesInordertofindappropriatewebexamples,ESL-WEPSgeneratesateachcheckpointasetofqueries.Thesequeriesinvolvethreediffer-entgranularitylevels,accordingtosentence’ssyntaxstructure:

1.ReducedSentenceLevel.Inordertousemorecontextualinformation,oursystempref-erentiallygeneratesamaximal-lengthqueryhereaftercalledS-Queries,byusingtheorigi-nalsentence.Forthecheckpointchunk,theverb/adj.tobecheckedisfoundandextractedbasedonPOStags;otherchunksaresimplyconcatenatedandusedtoformulatethequery.Forexample,forthefirstexampleinTable3,theS-Queryis[‘Ihave’AND‘thispersonfor

years’AND‘recognized’].

2.ChunkLevel.ThesystemsegmentseachESLsentenceaccordingtochunktagsandutilizeschunkpairstogenerateaquery,hereafterre-ferredtoasaC-Query,e.g.theC-QueryforthesecondexampleinTable3is[‘I’AND‘went’AND‘toclimb’AND‘atallmountain’AND‘lastweek’]3.WordLevel.Thesystemgeneratesqueriesbyusingkeywordsfromtheoriginalstring,intheprocessingeliminatingstopwordsusedintyp-icalIRengines,hereafterreferredtoasaW-Query,e.g.W-QueryforthefirstexampleinTable3is[‘I’AND‘have’AND‘person’AND‘years’AND‘recognized’]Asqueriesgetlonger,websearchenginestendtore-turnfewerandfewerresults.Therefore,ESL-WEPSfirstsegmentstheoriginalESLsentencebyusingpunctuationcharacterslikecommasandsemicolons,thengeneratesaqueryfromonlythepartwhichcon-tainsthegivencheckpoint.Whencheckingdeter-minerusage,threedifferentcases(aoran/the/none)

ParsedESLsentence

(NPI/PRP)(VPhave/VBPrecognized/VBN)(NPthis/DTperson/NN)(PPfor/IN)(NPyears/NNS)./.(NPI/PRP)(VPwent/VBD)(VPto/TOclimb/VB)(NPa/DTtall/JJmountain/NN)(NPlast/JJweek/NN)./.(NPI/PRP)(VPwent/VBD)(PPto/TO)(NPcoffee/NN)(NPshop/NN)(NPyesterday/NN)./.(NPSomeone/NN)(ADVPonce/RB)(VPsaid/VBD)(SBARthat/IN)(ADVPwhen/WRB)(NPyou/PRP)(VPmeet/VBP)(NPa/DTright/JJperson/NN)(PPat/IN)(NPthe/DTwrong/JJtime/NN),/,(NPit/PRP)(VP’s/VBZ)(NPa/DTpity/NN)./.

ErrorType

VNCANCDeterminerusageDeterminerusageCheckPoints

recognizedthispersontallmountain,lastweekcoffee,shop,yesterdaymeetarightpersonatthewrongtime

’sapity

Table3:ParsedESLsentencesandCheckPoints.

areconsideredforeachcheckpoint.Forinstance,giventhelastexampleinTable3,threeC-Querieswillbegenerated:[meetarightperson],[meettherightperson]and[meetrightperson].NotethatatermwhichhasbeenPOS-taggedasNNP(propernoun)willbeskippedandnotusedforgeneratingqueriesinordertoobtainmorewebhits.

RetreiveWebStatistics,CollectSnippetsTocol-lectenoughwebexamples,threelevelsofquerysetsaresubmittedtothesearchengineinthefollowingorder:S-Query,C-Query,andfinallyW-Query.Foreachquery,thewebhitsdfreturnedbysearchen-gineisrecorded,andthesnippetsfromthetop1000hitsarecollected.Forefficiencyreasons,wefollowDumais(2002)’sapproach:thesystemreliesonlyonsnippetsratherthanfull-textofpagesreturnedforeachhit;anddoesnotrelyonparsingorPOS-taggingforthisstep.However,alexiconisusedinordertodeterminethepossibleparts-of-speechofawordaswellasitsmorphologicalvariants.Forex-ample,tofindthecorrectVNCforagivennoun‘tea’inthereturnedsnippets,theverbdrankinthesameclausewillbematchedbefore‘tea’.

IdentifyErrorsandMineCorrectUsagesTode-tectdeterminerusageerrors,boththewebhitdfqandthelengthlqofagivenqueryqareutilized,sincelongerqueryphrasesusuallyleadtofewerwebhits.DFLq,DFLMAX,qmaxandRqaredefinedas:

DFLq=dfq×lq,foragivenqueryq;

DFLMAX=max(DFLq),qmax=argmax(DFLq),

q

q∈{queriesforagivencheckpoint};Rq=DFLq/DFLMAX,givenqueryqandcheckpoint.

IfDFLMAXislessthanagiventhresholdt1,thischeckpointwillbeskipped;otherwisetheqmaxin-dicatesthebestusage.WealsocalculatetherelativeratioRqforthreeusages(aoran/the/none).IfRqislargerthanathresholdt2foraqueryq,thesystemwillnotreportthatusageasanerrorbecauseitissufficientlysupportedbywebdata.Forcollocationcheckpoints,ESL-WEPSmayin-teracttwicewiththesearchengine:first,itissuesquerysetstocollectwebexamplesandidentifyplau-siblecollocationerrors;then,iferrorsaredetected,newquerysetswillbeissuedinthesecondstepinordertominecorrectcollocationsfromnewwebex-amples.Forexample,forthefirstsentenceinTa-ble3,theS-Querywillbe[‘Ihave’AND‘thisper-sonforyears’AND‘recognized’];thesysteman-alyzesreturnedsnippetsandidentifies‘recognized’asapossibleerror.ThesystemthenissuesanewS-Query[‘Ihave’AND‘thispersonforyears’],andfinallyminesthenewsetofsnippetstodiscoverthat‘known’isthepreferredlexicaloption.Incontrasttoproofingdeterminerusageserrors,mfreq:

mfreq=frequencyofmatchedcollocationalverb/adj.inthesnippetsforagivennoun,

isutilizedtobothidentifyerrorsandsuggestcorrectVNCs/ANCs.Ifmfreqislargerthanathresholdt3,thesystemwillconcludethatthecollocationisplausibleandskipthesuggestionstep.

4Experiments

Inordertoevaluatetheproofingalgorithmdescribedabove,weutilizedtheMSNsearchengineAPIandtheESLwritingsamplesetdescribedinSection1.1toevaluatethealgorithm’sperformanceontwotasks:determinerusageandVNCproofing.Fromapracticalstandpoint,weconsiderprecisionontheproofingtasktobeconsiderablymoreimportantthanrecall:falseflagsareannoyingandhighlyvis-ibletotheuser,whilerecallfailuresaremuchless

problematic.GiventhecomplicatednatureoftheESLerror

proofingtask,about60%ofESLsentencesinoursetthatcontaineddeterminererrorsalsocontainedothertypesofESLerrors.Asaresult,wewereforcedtoslightlyrevisethetypicalprecision/recallmea-surementinordertoevaluateperformance.First,

Errorsentence1proofingsuggestionErrorsentence2proofingsuggestionErrorsentence3nativespeakersuggestionsystemsuggestionErrorsentence4nativespeakersuggestionsystemsuggestionGoodProofingExamplesInmyopinion,therefore,whenwedescribeterrorism,itscruciallyimportantthatweconsiderthedegreeoftheinfluence(i.e.,power)ontheothercountries.considerthedegreeofinfluenceSomeoneoncesaidthatwhenyoumeetarightpersonatthewrongtime,it’sapity.meettherightpersonatthewrongtimePlausibleUsefulProofingExamplesThemostpowerfulplaceinBeijing,andinthewholeChina.inthewholeofChinainwholeChinaMe,Iwannakeepintouchwitholdfriendsandwannatalkwithanyonewhohasdifferentthought,etc.hasdifferentideashasadifferentthoughtTable4:ESLDeterminerUsageProofingbyNativeSpeakerandESL-WEPS.

Errorsentence1proofingsuggestionErrorsentence2proofingsuggestionErrorsentence3nativespeakersuggestionsystemsuggestionGoodProofingExamplesIhadgreattimethereandgotmanyfriends.mademanyfriendsWhichteamcantakethechampion?winthechampionPlausibleUsefulProofingExamplesItmaysoundsfunifIsaymyfirmresolutionofthisyearistogetagirlfriend.soundfunnymake*funorget*funTable5:ESLVNCProofingbyNativeSpeakerandESL-WEPS.

weconsideredthreecases:(1)thesystemcorrectlyidentifiesanerrorandproposesasuggestionthatex-actlymatchesthenativespeaker’srewrite;(2)thesystemcorrectlyidentifiesanerrorbutmakesasug-gestionthatdiffersfromthenativespeaker’sedit;and(3)thesystemincorrectlyidentifiesanerror.Inthefirstcase,weconsidertheproofinggood,inthesecond,plausiblyuseful,andinthethirdcaseitissimplywrong.Correspondingly,weintroducethecategoriesGoodPrecision(GP),PlausiblyUsefulPrecision(PUP)andErrorSuggestionRate(ESR),whichwerecalculatedby:

ofGoodProofings

GP=##;ofSystem󰀁sProofings#ofPlausiblyUsefulProofings

PUP=;#ofSystem󰀁sProofings#ofWrongProofings

ESR=#ofSystem󰀁sProofings;GP+PUP+ESR=1

becausesomeerrorstypesareoutofcurrentsys-tem’scheckingrange.

Thesystemachievedverypromisingperformancedespitethefactthatmanyofthetestsentencescon-tainedother,complexESLerrors:usingappro-priatesystemparameters,ESL-WEPSshowedre-call40.7%ondeterminerusageerrors,with62.5%oftheseproofingsuggestionsexactlymatchingtherewritesprovidedbynativespeakers.Crucially,thefalseflagratewasonly2%.Notethatarandom-guessingbaselinewasabout5%precision,7%re-call,butmorethan80%falseflagrate.

Forcollocationerrors,wefocusedonthemostcommonVNCproofingtask.mfreqandthresholdt3describedinSection3areusedtocontrolfalsealarm,GPandrecall.Asmallert3canreducerecall,butcanincreaseGP.Table7showshowperformancechangeswithdifferentsettingsfort3,andFig.2(b)plotstheGP/recallcurve.Resultsarenotverygood:asrecallincreases,GPdecreasestooquickly,sothatat30.7%recall,precisionisonly37.3%.Weat-tributethistothefactthatmostsearchenginesonlyreturnthetop1000websnippetsforeachqueryandourcurrentsystemreliesonthislimitednumberofsnippetstogenerateandrankcandidates.

Furthermore,assumingthatthereareoverallNAer-rorsforagiventypeAofESLerror,thetypicalrecallandfalsealarmwerecalculatedby:

Proofings

;recall=#ofGood

NA

#ofWrongProofings

falsealarm=#ofCheckpointsforESLerrorATable4andTable5showexamplesofGoodorPlausiblyUsefulproofingfordeterminerusageandcollocationerrors,respectively.Itcanbeseenthesystemmakesplausiblyusefulproofingsuggestions

Recall16.3%30.2%40.7%44.2%47.7%50.0%GP73.7%70.3%62.5%56.7%53.3%52.4%PUP15.8%16.2%25.0%29.9%29.9%29.3%falsealarm0.4%1.4%2.0%2.6%3.7%4.3%Table6:Proofingperformanceofdeterminerusagechangeswhensettingdifferentsystemparameters.

Recall11.3%12.9%17.8%25.8%29.0%30.7%GP77.8%53.3%52.4%43.2%40.9%37.3%PUP11.11%33.33%33.33%45.10%48.65%50.00%falsealarm0.28%0.57%0.85%0.85%1.13%2.55%Table7:VNCProofingperformancechangeswhensettingdifferentsystemparameters.

5Conclusion

Thispaperintroducedanapproachtothechalleng-ingreal-worldESLwritingerrorproofingtaskthatusestheindexofawebsearchengineforcor-pusstatistics.WevalidatedESL-WEPSonaweb-crawledESLwritingcorpusandcomparedthesys-tem’sproofingsuggestionstothoseproducedbyna-tiveEnglishspeakers.Promisingperformancewasachievedforproofingdeterminererrors,butlessgoodresultsforVNCproofing,possiblybecausethecurrentsystemuseswebsnippetstorankandgener-atecollocationcandidates.Wearecurrentlyinvesti-gatingamodifiedstrategythatexploitshighqualitylocalcollocation/synonymliststolimitthenumberofproposedVerb/Adj.candidates.

WearealsocollectingmoreESLdatatovalidateoursystemandareextendingoursystemtomoreESLerrortypes.RecentexperimentsonnewdatashowedthatESL-WEPScanalsoeffectivelyproofincorrectchoicesofprepositions.Laterresearchwillcomparetheweb-basedapproachtoconventionalcorpus-basedapproacheslikeGamonetal.(2008),andexploretheircombinationtoaddresscomplexESLerrors.

Good Precision vs. Recall󰀀Good Precision vs. Recall󰀀80.0%󰀀90.0%󰀀70.0%󰀀80.0%󰀀60.0%󰀀70.0%󰀀50.0%󰀀60.0%󰀀40.0%󰀀50.0%󰀀40.0%󰀀30.0%󰀀30.0%󰀀20.0%󰀀20.0%󰀀10.0%󰀀20.0%󰀀30.0%󰀀40.0%󰀀50.0%󰀀60.0%󰀀70.0%󰀀80.0%󰀀5.0%󰀀10.0%󰀀15.0%󰀀20.0%󰀀25.0%󰀀30.0%󰀀35.0%󰀀(a)Determiner Usage Error Proofing󰀀(b)VNC Error Proofing󰀀Figure2:GP/recallcurves.XandYaxisdenotesGPandRecallrespectively.

AcknowledgementTheauthorshavebenefitedextensivelyfromdiscussionswithMichaelGamonandChrisBrockett.WealsothanktheButlerHillGroupforcollectingtheESLexamples.

References

C.Brockett,W.B.Dolan,andM.Gamon.2006.Cor-rectingESLerrorsusingphrasalsmttechniques.InProceedingsofthe21stInternationalConferenceonComputationalLinguisticsandthe44thannualmeet-ingoftheACL,pages249–256,Sydney,Australia.S.Dumais,M.Banko,E.Brill,J.Lin,andA.Ng.2002.Webquestionanswering:ismorealwaysbetter?InProceedingsofthe25thAnnualInternationalACMSI-GIR,pages291–298,Tampere,Finland.M.Gamon,J.F.Gao,C.Brockett,A.Klementiev,W.B.Dolan,andL.Vanderwende.2008.UsingcontextualspellertechniquesandlanguagemodelingforESLer-rorcorrection.InProceedingsofIJCNLP2008,Hy-derabad,India,January.S.GuiandH.Yang,2003.ZhongguoXuexizheYingyuYuliaoku.(ChineseLearnerEnglishCorpus).Shang-haiWaiyuJiaoyuChubanshe,Shanghai.(InChinese).Jia-YanJian,Yu-ChiaChang,andJasonS.Chang.2004.TANGO:bilingualcollocationalconcordancer.InProceedingsoftheACL2004,pages19–23,Barcelona,Spain.D.LonsdaleandD.Strong-Krause.2003.AutomatedratingofESLessays.InProceedingsoftheNAACL-HLT03workshoponBuildingeducationalapplica-tionsusingnaturallanguageprocessing,pages61–67,Edmonton,Canada.G.Minnen,F.Bond,andA.Copestake.2000.Memory-basedlearningforarticlegeneration.InProceedingsoftheFourthConferenceonComputationalNaturalLanguageLearningandoftheSecondLearningLan-guageinLogicWorkshop,pages43–48.E.TjongKimSangandS.Buckholz.2000.Introductiontotheconll-2000sharedtask:Chunking.InProceed-ingsofCoNLL-2000andLLL-2000,pages127–132,Lisbon,Portugal.D.SchneiderandK.F.McCoy.1998.Recognizingsyn-tacticerrorsinthewritingofsecondlanguagelearn-ers.InProceedingsofthe17thinternationalconfer-enceonComputationallinguistics,pages1198–1204,Montreal,Quebec,Canada.C.-C.SheiandH.Pain.2000.Aneslwriter’scollo-cationalaid.ComputerAssistedLanguageLearning,13(2):167–182.