is anyone aware if this project is already being done by someone?
if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.
a couple initial impressions:.
@Django_Unchained:
No, its not that simple. I can extract the text, and I can even get that nasty middle column out of there. If I convert to an HTML format first, the "Sect" <div> tags hold the column information. But there are several other problems after that.
1) Sometimes words are joined together.
2) The cross reference symbols are letter characters so that we end up with many words spelled incorrectly.
MMM
is anyone aware if this project is already being done by someone?
if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.
a couple initial impressions:.
@trackregister99:
I can try to do a dump from my database to a series of text files for just the old version. I do not have the new version. So far the WT has only released this in a PDF version. If you go back you can see there are some difficulties extracting the text from the PDF. So as of now, there is nothing to compare the old version with...
MMM
is anyone aware if this project is already being done by someone?
if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.
a couple initial impressions:.
@trackregister99:
The web version that I parsed through and recorded (from jw.org, not the PDF) for the "old text", the text before the current 2013 version, has "all [other] things".
MMM
is anyone aware if this project is already being done by someone?
if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.
a couple initial impressions:.
@disfellowshipped1:
Looks like others want to know the differences between NWT and NWTv2 as well. We hope to document every change by way of a custom computer program and then develop some meaningful stats. Although, that part is still pending :)
MMM
is anyone aware if this project is already being done by someone?
if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.
a couple initial impressions:.
Apognophos,
No problem. sleep++;
PDF is a strange format. You can have text fields that "float". For example, if you want to display a table in PDF, you normally just position the text fields where you want, in the shape of a grid and then overlay lines to make it look like a grid. Something very odd is going on with that PDF when it comes to the acrossthe portion. Perhaps that is one text field that spans two lines with no whitespaces in there. When the extraction algorithm come across it, it looks like a single word. Note, I had Adobe Pro export it to HTML and it even caused the word to join. To repeat for emphasis - the HTML in my last screen shot was generated by extracting the page and pushing it through Adobe Pro itself. That is, Adobe is even confused when it tries to export as text.
When you have just HTML from the WT site, you know its all in the right order and the whitespace is ok because otherwise it wouldn't render right on their website. I think the harder task is that the cross reference symbols are plain letters. When you look at PDF as just text (just the text of the text fields), you loose the formatting - makes it look like the cross ref symbols are part of the words. Now there may be ways around all of this that I am not aware of yet.... who knows. Maybe we'll get lucky and the WT will put out a web version very soon.
As to your last statement: yes... a winmerge like format of the differences would essentially be reproducing the text of both editions into one file. Now if you came up with a category for a change, and could objectively categorize the changes and just gather stats - that, of course, is perfectly fine to pass around.
MMM
is anyone aware if this project is already being done by someone?
if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.
a couple initial impressions:.
zound,
I was thinking the same thing. I have a feeling that if I go forward with the PDF parse, we are going to end up with a bunch of errors that we won't know about until we examine each scripture. For each verse we would be asking, "Did that word bump up against the middle column so that the next word is actually placed right next to it without a whitespace?" When I pulled the text from the web site, I know that the spacing is right because it's needed to format correctly on the page. But for a PDF file, you can float that text anywhere.
MMM
is anyone aware if this project is already being done by someone?
if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.
a couple initial impressions:.
And one more update. Here is what it looks like when the page bytes are extracted, and sent back through Adobe Pro and exported as HTML. The export is a little better, and you get to see some inner tags. For example, the <div> tags with class "Sect" are the paragraphs. The great thing here is that you can easily peel out the middle column. Still the bad thing is that the column markers are letters, creating spelling mistakes. Also, notice that because PDFs are floating text for the most part, the extraction sometimes gets it wrong when it comes to word breaks. You can see it circled in red below... kinda stinks. I would much rather pull it from the web, you get the text all nice and neat....
MMM
is anyone aware if this project is already being done by someone?
if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.
a couple initial impressions:.
And here is another extraction method... A little less errors, but the text is draw out on the vertical.
GENESIS
1:20–2:520
Then
God
said:
“Let
the
waters
swarm
with
living
creatures,
1
and
let
flying
creatures
fly
above
the
earth
acrossthe
expanse
of
the
heavens.”a
21
And
God
created[SNIP FOR LENGTH - IT JUST GOES ON THAT WAY]
and
no
vegetation
of
the
field
had
begun
sprouting,
because
Jehovah
God
had1:30
1
Or
“life
as
a
soul;
a
living
soul.”
2:1
1
Lit.,
“and
all
their
army.”
2:2
1
Or
“making.”
2:4
1
The
first
occurrence
of
God’s
distinctive
personal
name,
(YHWH).
See
App.
A4.
MMM
is anyone aware if this project is already being done by someone?
if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.
a couple initial impressions:.
Apognophos,
As an update, I am able to pull text from the PDF file, but there are some odd anomalies. Here is what page 44 of the PDF looks like (I chose this page becuase it is the first full page of biblical text, after all that garbage they put at the front of the book - 42 pages of propaganda art BLA):
--------
GENESIS 1:20–2:5
20 Then God said: “Let the waters swarm with living creatures,1 and let .ying creatures .y above the earth acrossthe expanse of the heavens.”2a 21 And God created the great sea creatures2 and all living creatures1 that move and swarm in the waters according to theirkinds and every winged .ying creature according to its kind. And God saw that it was good. 22 With that God blessed them,saying: “Be fruitful and becomemany and .ll the waters of the sea,b and let the .ying creatures become many in the earth.”23 And there was evening and there was morning, a .fth day.
24 Then God said: “Let the earth bring forth living creatures1 according to their kinds, domestic animals and creeping animals2 and wild animals of the earth according to their kinds.”c And it was so. 25 And God went on to make the wild animals of the earth according totheir kinds and the domestic animals according to their kinds and all the creeping animals of the ground according to their kinds. And God saw that it was good.
26 Then God said: “Let usd make man in our image,e according to our likeness,f and let them have in subjection the .sh of the sea and the .ying creaturesof the heavens and the domestic animals and all the earth and every creeping animal that is moving on the earth.”g 27 And God went on to create the man in his image, in God’s image hecreated him; male and female hecreated them.h 28 Further, God blessed them, and God said to
1:20, 21, 24 1 Or “souls.” 1:20 2 Or “sky.” 1:21 2 Or “monsters.” 1:24 2 Or “moving animals,” apparently includingreptiles and forms of animal life di.erent from the other categories.CHAP. 1
a Ge 2:19
b Ne 9:6 Ps 104:25
c Ge 2:19
d Pr 8:30 Joh 1:3 Col 1:16
e 1Co 11:7
f Ge 5:1 Jas 3:9
g Ge 9:2
h Ps 139:14 Mt 19:4 Mr 10:6 1Co 11:7, 9
Second Col.
a Ge 9:1
b Ge 2:15
c Ps 8:4, 6
d Ge 9:3 Ps 104:14 Ac 14:17
e Ps 147:9 Mt 6:26
f De 32:4 Ps 104:24 1Ti 4:4
CHAP. 2
g Ne 9:6 Ps 146:6
h Ex 31:17 Heb 4:4
i Isa 45:18
them: “Be fruitful and become many, .ll the eartha and subdue it,b and have in subjectionc the .sh of the sea and the .ying creatures of the heavens andevery living creature that is moving on the earth.”
29 Then God said: “Here have given to you every seed-bearing plant that is on the entire earth and every tree with seed-bearing fruit. Let them serve as food for you.d 30 And to every wild animal of the earthand to every .ying creature of the heavens and to everything moving on the earth in which there is life,1 I have given all green vegetation for food.”e And it was so.
31 After that God saw everything he had made, and look! it was very good.f And there was evening and there was morning,a sixth day.
2
Thus the heavens and the earth and everything in them1 were completed.g 2 And by the seventh day, God had completed the work that he hadbeen doing,1 and he began to rest on the seventh day from allhis work that he had been doing.1h 3 And God went on to bless the seventh day and to declare it sacred, for on it God hasbeen resting from all the work that he has created, all that hepurposed to make.4 This is a history of the heavens and the earth in the time they were created, in the daythat Jehovah1 God made earth and heaven.i
5 No bush of the .eld was yet on the earth and no vegetation of the .eld had begun sprouting, because Jehovah God had
1:30 1 Or “life as a soul; a living soul.”
2:1 1 Lit., “and all their army.” 2:2 1 Or “making.” 2:4 1 The .rst occurrence of God’s distinctive personal name, 565 (YHWH). See App. A4.--------
The first thing you may notice is that the cross reference markers are going to make it very difficult. The web version used "+" and "*" hyperlinks. The PDF version is like the printed version in the sense that it uses normal letters, causing odd misspellings to occur. You can see the middle column lines too. But, as you may know, a human looking onto text like this is one thing, but developing an algorithm to intelligently parse this is another story. I'll have to work at it. The cross reference letters in the main text seem like the largest problem so far - oh, and the footnotes merge into the main text. That's the issue with PDFs, they are basically text positioned around a canvas. You don't get line breaks you would expect.
Edit: also note how "flying" didn't render out completely.
MMM
is anyone aware if this project is already being done by someone?
if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.
a couple initial impressions:.
Apognophos,
Just an update: the web parse completed successfully. There are 1189 distinct book/chapter combos and 31,114 verse records total.
Now onto the PDF file.
MMM
