3

Сиквел "read_table обрезает предложения" ...

Есть текстовый файл вида:

date : 2018-03-08T12:56:30+00:00
content : [A very long string,
which might contain multiple line breaks]
href : https://www.google.com/

date : 2018-03-08T12:56:30+00:00
content : [Another long string ...]
href : https://www.google.com/

...

Как распарсить его в DataFrame с тремя столбцами: date, content, url?

2 ответа 2

4

Можно упростить регулярное выражение:

>>> re.findall(r'(?ms)' + '\s*'.join([ # allow whitespace between tokens
...      '^date', ':', '(\S+)',        # no whitespace in the timestamp
...      '^content', ':', '(.*?)',     # multiline non-greedy text until ^href
...      '^href', ':', '(\S+)']),      # no whitespace in the url
...            text)
[('2018-03-08T12:56:30+00:00', '[A very long string,\nwhich might contain multiple line breaks]', 'https://www.google.com/'),
 ('2018-03-08T12:56:30+00:00', '[Another long string ...]', 'https://www.google.com/')]
4

Код:

import re
from pathlib import Path
import pandas as pd

fn = r'C:\download\textout.txt'

text = Path(fn).read_text(encoding='utf-8')

pat = r'date\s*:\s*([^\r\n]*)[\r\n]*content\s*:\s\[([^\]]*)\][\r\n]*href\s*:\s*([^\r\n]*)'

df = pd.DataFrame(re.findall(pat, text, flags=re.S|re.M), columns=['date','content','url'])
df['date'] = pd.to_datetime(df['date'])

Пример полученного DataFrame:

In [26]: df
Out[26]:
                    date                                            content  \
0    2018-03-08 12:56:30  The price of cryptocurrencies across all marke...
1    2018-03-08 12:32:04  GMO Internet has released a monthly report on ...
2    2018-03-08 11:35:00  Despite all its previous efforts to prevent th...
3    2018-03-08 09:05:38  On Wednesday, March 7, the U.S. regulator the ...
4    2018-03-08 07:50:00  This week the popular wallet provider Bread (B...
5    2018-03-08 04:00:43  This month a law firm called Polsinelli LLP pu...
6    2018-03-08 01:30:27  Binance has found itself at the center of an u...
...                  ...                                                ...
2043 2017-04-04 06:00:41  Kim Dotcom has recently tweeted a preview of t...
2044 2017-04-04 03:00:13  On August 2, 2016, the leading bitcoin exchang...
2045 2017-04-03 19:00:18  This past weekend on April 1-2 in Berlin, Germ...
2046 2017-04-01 19:00:04  People often cast nasty judgment on Bitcoin. T...
2047 2017-04-01 14:00:53  This past week on March 24 the San Francisco-b...
2048 2017-03-31 12:38:51  This week the price of bitcoin has remained fa...
2049 2017-03-31 06:00:31  What is the legality of Bitcoin in India? This...

                                                    url
0     https://news.bitcoin.com/markets-update-crypto...
1     https://news.bitcoin.com/japanese-conglomerate...
2     https://news.bitcoin.com/chinese-internet-regu...
3     https://news.bitcoin.com/sec-publishes-warning...
4     https://news.bitcoin.com/wallet-provider-bread...
5     https://news.bitcoin.com/research-paper-says-i...
6     https://news.bitcoin.com/bots-blamed-for-binan...
...                                                 ...
2043  https://news.bitcoin.com/kim-dotcom-bitcache-m...
2044  https://news.bitcoin.com/bitfinex-bfx-tokens-r...
2045  https://news.bitcoin.com/new-alliances-at-bitc...
2046  https://news.bitcoin.com/bitcoin-used-buy-sex-...
2047  https://news.bitcoin.com/fifty-developers-hack...
2048  https://news.bitcoin.com/markets-update-bitcoi...
2049  https://news.bitcoin.com/bitcoin-legality-in-i...

[2050 rows x 3 columns]

In [27]: df.dtypes
Out[27]:
date       datetime64[ns]
content            object
url                object
dtype: object

Ваш ответ

Нажимая «Отправить ответ», вы соглашаетесь с условиями пользования и подтверждаете, что прочитали политику конфиденциальности.

Всё ещё ищете ответ? Посмотрите другие вопросы с метками или задайте свой вопрос.