# Transformer-self

**Repository Path**: firslov/transformer-self

## Basic Information

- **Project Name**: Transformer-self
- **Description**: Transformer reproduction.
- **Primary Language**: Python
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2022-04-30
- **Last Updated**: 2022-07-11

## Categories & Tags

**Categories**: Uncategorized

**Tags**: Python, Deep-learning, Computer-vision

## README

# Transformer-self

## 导入包

```python
from pyitcast.transformer_utils import Batch
from pyitcast.transformer_utils import get_std_opt
from pyitcast.transformer_utils import LabelSmoothing
from pyitcast.transformer_utils import SimpleLossCompute
from pyitcast.transformer_utils import run_epoch
from pyitcast.transformer_utils import greedy_decode
import torch
import numpy as np
```

```python
from transformer import Transformer
```

## copy任务

```python
def data_generator(vocab, num_batch, batch_size):
    for i in range(num_batch):
        data = torch.from_numpy(np.random.randint(1, vocab, size=(batch_size, 10)))
        data[:, 0] = 1
        source = data.requires_grad_(False)
        target = data.requires_grad_(False)
        
        yield Batch(source, target)
```

```python
vocab = 11
model = Transformer(vocab, vocab, N=2)
model_optimizer = get_std_opt(model)
criterion = LabelSmoothing(size=vocab, padding_idx=0, smoothing=0.0)
loss = SimpleLossCompute(model.generator, criterion, model_optimizer)
```

```python
def run(model, loss, epochs=20):
    for epoch in range(epochs):
        # train mode
        model.train()
        run_epoch(data_generator(vocab, 20, 15), model, loss)
        # eval mode
        model.eval()
        run_epoch(data_generator(vocab, 5, 15), model, loss)
        
    model.eval()
    source = torch.LongTensor([[1, 3, 2, 5, 4, 6, 7, 9, 8, 10]])
    source_mask = torch.ones(1, 1, 10)
    result = greedy_decode(model, source, source_mask, max_len=10, start_symbol=1)
    print(result)
```

## 运行效果

```python
run(model, loss)
```

```shell
Epoch Step: 1 Loss: 3.397329 Tokens per Sec: 1244.841431
Epoch Step: 1 Loss: 2.777961 Tokens per Sec: 1656.990845
Epoch Step: 1 Loss: 2.888927 Tokens per Sec: 1413.374512
Epoch Step: 1 Loss: 2.186104 Tokens per Sec: 1475.620605
Epoch Step: 1 Loss: 2.382907 Tokens per Sec: 1339.549805
Epoch Step: 1 Loss: 1.871710 Tokens per Sec: 1609.749634
Epoch Step: 1 Loss: 1.934245 Tokens per Sec: 1446.588867
Epoch Step: 1 Loss: 1.801598 Tokens per Sec: 1591.567139
Epoch Step: 1 Loss: 1.919525 Tokens per Sec: 1437.760498
Epoch Step: 1 Loss: 1.563237 Tokens per Sec: 1595.322632
Epoch Step: 1 Loss: 1.757882 Tokens per Sec: 1303.768066
Epoch Step: 1 Loss: 1.577413 Tokens per Sec: 1315.841675
Epoch Step: 1 Loss: 1.678141 Tokens per Sec: 1374.695923
Epoch Step: 1 Loss: 1.303878 Tokens per Sec: 1601.366089
Epoch Step: 1 Loss: 1.431116 Tokens per Sec: 1426.736450
Epoch Step: 1 Loss: 1.219834 Tokens per Sec: 1668.344238
Epoch Step: 1 Loss: 1.452936 Tokens per Sec: 1526.743652
Epoch Step: 1 Loss: 0.956609 Tokens per Sec: 1672.920898
Epoch Step: 1 Loss: 1.229171 Tokens per Sec: 1420.073853
Epoch Step: 1 Loss: 0.852122 Tokens per Sec: 1607.389404
Epoch Step: 1 Loss: 1.153687 Tokens per Sec: 1309.262207
Epoch Step: 1 Loss: 0.689962 Tokens per Sec: 1468.972900
Epoch Step: 1 Loss: 0.953595 Tokens per Sec: 1352.048218
Epoch Step: 1 Loss: 0.489096 Tokens per Sec: 1523.848999
Epoch Step: 1 Loss: 0.732036 Tokens per Sec: 1437.125488
Epoch Step: 1 Loss: 0.331112 Tokens per Sec: 1617.987915
Epoch Step: 1 Loss: 0.737162 Tokens per Sec: 1517.093994
Epoch Step: 1 Loss: 0.246102 Tokens per Sec: 1785.905151
Epoch Step: 1 Loss: 0.638294 Tokens per Sec: 1405.962524
Epoch Step: 1 Loss: 0.143088 Tokens per Sec: 1734.285034
Epoch Step: 1 Loss: 0.495012 Tokens per Sec: 1464.080811
Epoch Step: 1 Loss: 0.133364 Tokens per Sec: 1668.511475
Epoch Step: 1 Loss: 0.407922 Tokens per Sec: 1511.243042
Epoch Step: 1 Loss: 0.092330 Tokens per Sec: 1763.786499
Epoch Step: 1 Loss: 0.211001 Tokens per Sec: 1415.754517
Epoch Step: 1 Loss: 0.124462 Tokens per Sec: 1767.039551
Epoch Step: 1 Loss: 0.325912 Tokens per Sec: 1438.587769
Epoch Step: 1 Loss: 0.055157 Tokens per Sec: 1775.018921
Epoch Step: 1 Loss: 0.279179 Tokens per Sec: 1414.850708
Epoch Step: 1 Loss: 0.072871 Tokens per Sec: 1682.429932
tensor([[ 1,  3,  2,  5,  4,  6,  7,  9,  8, 10]])
```